Science is based on experimental testing and data. Yet ironically the publishing of science was until recently based on faith not study. BMJ has a vision of becoming the world's first evidence based publisher. We have been publishing around evidence based medicine for some years and hope that the same thinking can transform publishing.
We carry out research into all aspects of BMJ's operation with the objective of developing, promoting, and incorporating evidence-based and responsible publishing into our own practices. Our research projects usually originate from within BMJ, but our work has a wider focus: nearly all of the issues we research have relevance for journal editors, authors, peer reviewers and publishers working across biomedical science. We also conduct collaborative research projects with researchers in international institutions and offer a PhD program on the responsible conduct of publishing scientific research. As a publisher of numerous journals we have a rich dataset of submitted articles and privileged contact with authors and reviewers worldwide.
Here you will find useful information on research undertaken at BMJ into all aspects of publishing science including peer review, conflicts of interests, editorial decision making, open access publishing, data sharing, authorship/contributorship, presentation of research, research ethics, publication bias, and education. Projects undertaken include randomised controlled trials of open peer review and the effectiveness of peer review training, author surveys on the use of statistical expertise in medical research, preferences for presentational style of scientific research, perceptions of electronic publishing, attitudes towards open access publishing, and reader surveys to understand the effect of competing interest statements on research credibility. If you have any ideas for collaborative research in this area or wish to pursue your own PhD, please contact us.
Sara Schroter, senior researcher, The BMJ
We conducted a randomised controlled trial to evaluate the effect on the quality of peer review of blinding reviewers to authors' identities and requiring reviewers to sign their reports. We found that neither blinding reviewers to the authors and origin of the paper nor requiring them to sign their reports had any effect on the rate of detection of errors. Such measures are unlikely to improve the quality of peer review reports.
Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA 1998: 280: 237-240. [Abstract]]
We conducted a randomised controlled trial to determine whether concealing authors' identities from reviewers (blinding) and/or revealing the reviewer's identity to a coreviewer (unmasking) affects the quality of reviews, the time taken to carry out reviews, and the recommendation regarding publication. We found that blinding and unmasking made no editorially significant difference to review quality, reviewers' recommendations, or time taken to review.
van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review: A randomized trial. JAMA 1998; 280: 234 – 7. [Abstract]
We conducted a study to determine the characteristics of reviewers for a general medical journal who produce high-quality reviews and to describe the characteristics of a good review, particularly in terms of the time spent reviewing and turnaround time. In a logistic regression analysis, the only significant factor associated with higher-quality ratings by both editors and authors was reviewers trained in epidemiology or statistics. Younger age also was an independent predictor for editors' quality assessments, while reviews performed by reviewers who were members of an editorial board were rated of poorer quality by authors. Review quality increased with time spent on a review, up to 3 hours but not beyond.
Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for a general medical journal? JAMA 1998; 280(3): 231 – 233. [Abstract]
We developed and validated an instrument to measure the quality of reviews. The Review Quality Instrument (RQI) assesses the extent to which a reviewer has commented on five aspects of a manuscript (importance of the research question, originality of the paper, strengths and weaknesses of the method, presentation, interpretation of results) and on two aspects of the review (constructiveness and substantiation of comments). The RQI’s internal consistency was high (Cronbach's alpha 0.84). The mean total score (based on the seven items each scored on a 5-point Likert scale from 1 to 5) had good test-retest (Kw = 1.00) and inter-rater (Kw = 0.83) reliability. There was no evidence of floor or ceiling effects, construct validity was evident, and the respondent burden was acceptable (2-10 minutes). The instrument can be recommended for use in the study of peer review and we have used it in several of our studies.
van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. Journal of Clinical Epidemiology 1999; 52(7): 625 – 629. [Abstract]
We conducted a randomised controlled trial to examine the effect on peer review of asking reviewers to have their identity revealed to the authors of the paper. We found that this had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review.
van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: A randomised trial. BMJ 1999; 318: 23 – 27. [Abstract] [Full Text]
Many studies have illustrated the inadequacies of peer review and its limitations in improving the quality of research papers. However, few studies have evaluated interventions that try to improve peer review, and there have been no previous randomised controlled trials to examine the effects of training. Training that would be feasible for reviewers to undergo and for a journal to provide would have to be short or provided at a distance. Although the effectiveness of short educational interventions is questionable, some brief interventions have been shown to be successful (depending on what is being taught and the methods used). We aimed to determine whether reviewers for The BMJ who underwent training would produce reviews of better quality than those who received no training; whether face to face training would be more beneficial than a self taught package; and whether any training effect would last at least six months. The training materials for the trial are now freely available on bmj.com.
Schroter S, Groves T. BMJ training for peer reviewers. BMJ 2004; 328. [Full Text]
White I, Carpenter J, Evans S, Schroter S. Eliciting and using expert opinions about dropout bias in randomised controlled trials. Clinical Trials: Journal of the Society for Clinical Trials 2007;4(2):125-39. [Abstract]
Schroter S, Black N, Evans S, Godlee F, Osorio L, Smith R. What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med 2008:101: 507–514. [Full Text]
Many journals give authors the opportunity to suggest reviewers to review their paper, but editors’ use of these reviewers vary as some are concerned that they may favour the author. We conducted a study comparing author- and editor-suggested reviewers of ten biomedical journals in a range of specialties to investigate differences in review quality and recommendation for publication. We found no difference in review quality but author-suggested reviewers tended to make more favourable recommendations for publication.
Schroter S, Tite L, Hutchings, A, Black N. Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. JAMA 2006;295:314-7. [Abstract]
Peer reviewers are usually unpaid and their efforts are not formally acknowledged. Editors of some journals experience difficulty finding appropriate reviewers who are able to complete timely reviews, resulting in publication delay. We conducted a survey of reviewers from five BMJ Journals to determine why reviewers decline to review and their opinions of various incentives. We found that reviewers are more likely to accept to review a manuscript when it is relevant to their area of interest and lack of time is the principal factor in the decision to decline. Most respondents agreed that financial incentives would not be effective when time constraints are prohibitive
Tite L, Schroter S. Why do reviewers decline to review for journals? A survey. J Epidemiol Community Health 2007;61:9-12. [Abstract]
We conducted an RCT to see whether telling peer reviewers that their signed reviews of original research papers might be posted on The BMJ's website would affect the quality of their reviews.
We found that this had no important effect on review quality. Although the possibility of posting reviews online was associated with a high refusal rate among potential peer reviewers and an increase in the amount of time taken to write a review, we believe that the ethical arguments in favour of open peer review more than outweigh these disadvantages.
van Rooyen S, Delamothe T, Evans SJW. Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ 2010;341:c5729 [Full Text]
Starling Group, co-chaired by Trish Groves (The BMJ) and Professor Liselotte Højgaard (European Medical Research Councils).
There is now a considerable body of evidence on the methods, outcomes, effectiveness, best practice, and ethics of editorial peer review of submissions to biomedical journals. In contrast, grant review, which is practised in a wide variety of ways, has a relatively sparse evidence base. It is possible that grant review might be informed by editorial peer review practice and evidence. The Starling Group is a newly set up international group interested in improving grant review for health and medical research. The Starling Group conducted two surveys. The first was an international survey of biomedical research funding organisations to measure typical workloads of funding organisations and current problems they face. The second was an international survey of external grant reviewers to measure typical workloads of external grant reviewers and factors that motivate them to review.
Schroter S, Groves T, Højgaard L. Surveys of current status in biomedical science grant review: funding organisations’ and grant reviewers’ perspectives. BMC Medicine 2010;8:62. [Full text PDF]
The “publish or perish” mantra is well established in academic circles, but do researchers have to sacrifice leisure time to achieve this ideal? We conducted an observational study of research manuscripts and peer reviews submitted to The BMJ and BMJ Open between 2012 and 2019 to determine whether researchers are submitting manuscripts and peer reviews out of hours and whether this has changed over time.
Barnett Adrian, Mewburn Inger, Schroter Sara. Working 9 to 5, not the way to make an academic living: observational analysis of manuscript and peer review submissions over time. BMJ 2019;367:I6460. [Full text PDF]
We performed a randomised controlled trial to evaluate the effect of an editorial intervention performed by a CONSORT expert on the completeness of RCT reporting. The intervention consisted of (i) assessing during peer review the consistency between the CONSORT checklist submitted by authors and the information reported in the manuscript, and (ii) asking authors for changes in relation to the inconsistencies identified. This study was conducted by David Blanco in collaboration with BMJ Publishing Group as part of the MIROR doctoral training programme funded by Marie Skłodowska-Curie Actions, dedicated to Methods in Research on Research (MIROR).
Blanco D, Schroter S, Aldcroft A, Moher D, Boutron I, Kirkham JJ, Cobo E. Effect of an editorial intervention to improve the transparency of randomised trials: a randomised controlled trial. [Paper in preparation].
Abstract accepted for presentation at the PEERE conference on peer review, Valencia, Spain, March 2020. [PDF]
Conflict of interest has been defined as a set of conditions in which professional judgment concerning a primary interest (such as patient welfare or the validity of research) can be influenced by a secondary interest (such as financial gain). Despite increasing evidence that conflict of interest influences authors' conclusions, there has been little research into the effect on readers' perceptions of research. We conducted two studies to see whether a declarations of competing interests influences readers' perceptions of the interest, importance, relevance, validity, and believability of a study.
Chaudhry S, Schroter S, Smith R, Morris J. Does declaration of competing interests affect reader perceptions: A randomised trial? BMJ 2002; 325:1391-2. [Full Text]
Schroter S, Morris J, Chaudhry S, Smith, R, Barratt H. Does the type of competing interest statement affect reader perceptions of the credibility of research? A randomised trial. BMJ 2004; 328:742-3. [Full Text]
Williams HC, Naldi L, Paul C, Vahlquist A, Schroter S, Jobling R. Conflicts of Interest in Dermatology. Acta Derm Venereol. 2006;86(6):485-97. [Abstract]
Two previous RCTs showed that readers perceptions of research were influenced by financial competing interests. In this trial we investigated how different competing interest statements affect clinical readers’ confidence in the article’s conclusions (primary outcome), its importance, their level of interest in the article, and their likelihood to change practice after reading it. We tested a range of financial COI statements that are commonly reported to journals and frequently occur in medical practice.
Schroter S, Pakpoor J, Morris J, Chew M, Godlee F. The effect of different financial competing interest statements on readers’ perceptions of clinical educational articles: Study protocol for a randomised controlled trial. BMJ Open 2016;6:e012677. [Protocol]
Schroter S, Pakpoor J, Morris J, Chew M, Godlee F. Effect of different financial competing interest statements on readers’ perceptions of clinical educational articles: A randomised controlled trial. BMJ Open 2019;9:e025029. [Full text]
The International Committee of Medical Journal Editors (ICMJE) recognises that all persons involved in peer review and journal publishing can have conflicting interests. Although authors are expected to declare conflicts of interest at the time of publication, there is no requirement for journal editors to publicly declare their own competing interests. We conducted a survey of editors of General and Internal Medicine journals to explore their policies regarding competing interests of editors and other journal staff.
Haivas I, Schroter S, Waechter F, Smith R. Editors' declaration of their own conflicts of interest. Canadian Medical Association Journal 2004; 171(5): 475-476. [Full Text PDF]
We investigated whether editors at The BMJ could make a decision about manuscripts based on reading only the abstract and how this initial decision differed from when they read the whole submission. Editors were able to make an editorial decision based on reading only the abstract for 62% (229/372) of submissions. Where a decision could be made, there was no significant difference between the two assessments in the proportion of papers rejected immediately by a single editor. Neither was there a significant difference between the two assessments in the proportion of papers where it was suggested that the submission should be sent to an external reviewer.
Since the introduction of the world wide web, peer-reviewed journal publishers have been wrestling with the dilemma of how to survive in a market place that facilitates and encourages the free exchange of electronic content. This web-based movement is in opposition to the traditional subscriptions-based business model of scientific publishing that relies heavily on print-based pricing and distribution. One emerging alternative to the traditional journal business model is Open Access publishing where lawful free access to journal content is provided online with production funded through means other than subscription charges. Various models for funding open access have been proposed, but one in particular has generated considerable interest and has become the business model for several new open access journals. The 'author-pays' model requires that authors, or their granting agencies or institutions, pay a contribution to the cost of publishing on submission and / or acceptance of their paper. We conducted some research with authors to help understand their perspectives. The research comprised several stages:
We conducted semi-structured telephone interviews to explore authors' attitudes towards open access publishing and author charges, their perceptions of journals that charge authors, and whether they would be willing to submit to these journals. We found authors were more aware of the concepts of open access publishing and author pays models than previously reported. Almost all authors supported the concept of open access, but few had submitted to an open access journal, other than The BMJ. Reasons for not submitting included lack of awareness of which journals publish with open access, and journal quality taking a higher priority in decision making than the availability of open access. Authors disliked the idea of author charges without institutional support and were concerned about implications for authors from developing countries and those without research funding.
Schroter S, Tite L, Smith R. Perceptions of Open Access Publishing: Interviews with Journal Authors. BMJ 2005;330:756-9. [Full Text]
We conducted an electronic survey of 468 journal authors to assess current knowledge and perceptions of open access and author-pays publishing. Compared with non-open access subscription-based journals, 35% agreed that open access author-pays journals have a greater capacity to publish more content making it easier to get published, 27% thought they had lower impact factors, 31% thought they had faster and more timely publications, and 46% agreed that people will think anyone can pay to get published. Over half thought they would have to make a contribution or pay the full cost of an author charge.
Schroter S, Tite L. Open access publishing and author-pays business models: A survey of authors’ knowledge and perceptions. J R Soc Med 2006;99:130-7.[Full Text PDF]
The author-pays model (open access publishing funded through author charges) is dependent on authors having access to financial support at the time their research papers are accepted. We conducted a survey of 377 authors from three journals in different specialties to determine the availability of external funding for publication charges at different points in the research process. We found a large proportion of published research is not externally funded, and many funded researchers do not have access to financial support at the time their paper is accepted for publication.
Schroter S, Tite L., Kassem A. Financial support at the time of paper acceptance: A survey of three medical journals. Learned Publishing 2006;19(4):291-7. [Full Text PDF]
We carried out a survey to determine whether free access to research articles on bmj.com is an important factor in authors' decisions on whether to submit to the The BMJ, whether the introduction of access controls to part of the The BMJ's content has influenced authors' perceptions of the journal, and whether the introduction of further access controls would influence authors' perceptions. We found authors value free access to research articles and consider this an important factor in deciding whether to submit to The BMJ. Closing access to research articles would have a negative effect on authors' perceptions of the journal and their likeliness to submit.
Schroter S. Importance of free access to research articles on decision to submit to the BMJ: A survey of authors. BMJ 2006;332:394-6. [Full Text]
We investigated the nature and frequency of statistician involvement in medical research and its relation to the final editorial decision. Authors submitting to The BMJ and Annals of Internal Medicine in 2001 were sent a questionnaire asking if they received assistance from a person with statistical expertise, the nature of any such contribution, and reasons why, if no statistical input was received. We found that statistical input to medical research is widely recommended but inconsistently obtained. Individuals providing such expertise are often not involved until the analysis of data and many go unrecognised by either authorship or acknowledgment.
It is important that authors understand why their work has been rejected by a journal as this may influence the publication, and whether and where they submit in future. Little research has been conducted into authors’ perceptions of editorial review. We conducted two cross-sectional surveys of The BMJ authors whose papers were rejected without external peer review. The purpose was to evaluate differences in perceptions of the review system according to method of rejection and to evaluate preferred methods of receiving rejection.
Barratt H, Schroter S, Smith R. Two postal surveys of different methods of communicating rejection to authors submitting to a general medical journal. Accountability in Research: Policies and Quality Assurance 2003;10(4):289 – 299. [Abstract]
Some journals allow authors to submit supplementary material for consideration when their main manuscript is going through the editorial decision-making process and for possible publication alongside the article, should it be accepted. This might include documents or other materials such as the original research protocols, data collection forms, questionnaires, additional data tables and figures, and interview transcripts. Many journals also require authors to provide some supplementary materials on submission, for example links to the study data, completed reporting guideline checklists and flow diagrams (such as CONSORT, STROBE, PRISMA, STARD, etc.). The preparation, consideration and publication of all this material requires the use of resources by the authors, peer reviewers and journal. Is this necessary? Is it wasteful? Is it useful? Who reads the various pieces of supplementary material and do they need to be accessible from the research article? In this survey we explore the value, use, and role of this material in biomedical journal articles from the perspectives of authors, peer reviewers and readers of 17 BMJ Publishing Group journals.
Schroter S, Price A, Clarke M. The role of supplementary material in journal articles: Surveys of authors, reviewers, and readers. BMJ Open 2018;8:e021753. doi: 10.1136/bmjopen-2018-021753. [Full text]
To be useable in clinical practise, treatments studied in trials must provide sufficient information to enable clinicians and researchers to replicate. In this collaborative study between The Centre of Evidence Based Medicine in Oxford (CEBM) and The BMJ, we assessed the completeness of treatment descriptions in published randomised controlled trials (RCTs) using a checklist and determined the extent to which peer reviewers and editors commented on the quality of reporting of treatments.
Schroter S, Glasziou P, Heneghan C. Quality of descriptions of treatments: A review of published randomised controlled trials. BMJ Open 2012;2:e001978. doi:10.1136/bmjopen-2012-001978. [Full text]
Tables and figures can convey details and complex relationships not easily described in text. We characterised the quantity and quality of data tables and figures in reports of randomised controlled trials (RCTs) submitted to The BMJ and published in peer-reviewed journals. We investigated how the peer review process affected table and figure quality. We found the numbers of tables and figures did not change markedly between submission and publication. Less than half the figures met their data presentation potential, with most failing to portray by-subject data and few displaying advanced features such as pairing, symbolic dimensionality, or small multiples. The BMJ external peer reviewers seldom commented on tables or figures.
Schriger DL, Sinha R, Schroter S, Liu PY, Altman DG. From Submission to Publication: A Retrospective Review of the Tables and Figures in a Cohort of Randomized Controlled Trials Submitted to the British Medical Journal. Ann Emerg Med 2006 Dec;48(6):750-6. [Abstract]
Editorial space is a scarce resource and editors struggle to meet the expectations of both authors and readers. Several scientific journals have adopted strategies to publish detailed article content on the web and a more concise presentation in the print journal. We elicited readers’ and authors’ preferences for three different presentational styles of abridged print versions: the conventional approach where the Introduction-Methods-Results-Discussion structure is maintained and the text simply shortened, a journalistic version with a very different structure, and a version with an enhanced abstract similar to those used in evidence-based medicine journals. We found authors and readers prefer a more conventional style of abridged papers.
We conducted a survey to evaluate how acceptable authors find the The BMJ's current practice of publishing short versions of research articles in the paper journal and a longer version on the web (ELPS) and to determine authors’ attitudes towards publishing only abstracts in the paper journal and publishing unedited versions on bmj.com once papers have been accepted for publication.
Considerable time is expended by journals investigating ethical transgressions and misconduct that may be caused by ignorance rather than wilful deceit. Publication ethics are rarely taught. We conducted an international survey of authors submitting to 20 BMJ journals to measure the level of awareness of key ethical issues in publishing amongst a large sample of currently active researchers. We hope the results stimulate discussion within journal publishing and prompt efforts to improve knowledge and, in turn, behaviour.
Schroter S, Roberts J, Loder E, Penzien DB, Mahadeo S, Houle T. Biomedical authors’ awareness of publication ethics: an international survey. BMJ Open2018;8:e021282. doi: 10.1136/bmjopen-2017-021282.[Full text]
This collaborative study with the Health Research Authority is one of the few controlled evaluations of a quality improvement intervention in research ethics review. It found that provision of an ethics officer role, as deployed in this project, does not shorten time to final decision by research ethics committees nor increase the proportion of applications that got a favourable opinion first time.
Dixon-woods M, Foy C, Hayden C, Al-Shahi Salman R, Tebbutt S, Schroter S. Can an ethics officer role reduce delays in research ethics approval? A mixed-method evaluation of an improvement project. BMJ Open 2016;6:e011973. [Full text]
Authors are required to describe in their manuscripts ethical approval from an appropriate committee and how consent was obtained from subjects when research involves human subjects. Previous studies have focused on concordance with these regulations within a single speciality or for clinical trials. We assessed reporting of these protections for several study designs published in Annals of Internal Medicine, The BMJ, JAMA, Lancet and New England Journal of Medicine. We found the reporting of ethical approval and consent has improved for RCTs but journals are less good at reporting this information for other study designs.
Schroter S, Plowman R, Hutchings A, Gonzalez A. Reporting of ethical committee approval and patient consent by study design in five general medical journals. J Med Ethics 2006;32:718-23. [Abstract] [Full text PDF]
While documented plans for patient and public involvement (PPI) in research are required in many grant applications, little is known about how frequently PPI occurs in practice. We conducted a before and after comparison of PPI reported in research papers published in The BMJ before and 1 year after the introduction of a journal policy requiring authors to report if and how they involved patients and the public within their papers.
Price A, Schroter S, Snow R, Hicks M, Harmston R, Staniszewska S, Parker S, Richards T. Frequency of reporting on Patient and Public Involvement (PPI) in research studies published in a general medical journal: A descriptive study. BMJ Open 2018;8:e020452. doi: 10.1136/bmjopen-2017-020452. [Full text]
In 2014/15 The BMJ and Research Involvement and Engagement (RIE) became the first journals to routinely include patients and the public in the peer review process of journal articles. In this collaborative survey carried out by The BMJ and RIE, with embedded patient involvement, we explore the early experiences of our patient and public reviewers. Reviewers were asked what motivates them to review research articles, if and why they have declined to review, their confidence in reviewing, their perceptions of open review, their satisfaction with the process, how they would like to be acknowledged for their contributions and their views on how the process could be improved. This knowledge will be used to inform evidence-based guidance for patient and public reviewers.
Schroter S, Price A, Flemyng E, et al. Perspectives on involvement in the peer-review process: surveys of patient and public reviewers at two journals. BMJ Open2018;8:0:e023357. doi:10.1136/bmjopen-2018-023357.[Full text]
Dissemination of research findings is central to research integrity and promoting discussion of new knowledge and its potential for translation into practice and policy. Dissemination is the process by which results of studies are communicated to the stakeholders of that research. Stakeholders can include health policy makers, academics, practicing clinicians and other healthcare workers, patients, the public, and research funders. However, effective communication and active dissemination of research results to patients, which is essential for them to make informed decisions and choices, is sometimes an afterthought among researchers. We surveyed researchers who had recently published a clinical trial to determine how frequently and in what format they disseminated the results of their clinical trials to trial participants (and patient groups), and to identify barriers to dissemination.
Schroter S, Price A, Malički M, et al Frequency and format of clinical trial results dissemination to patients: a survey of authors of trials indexed in PubMed. BMJ Open2019;9:e032701. doi: 10.1136/bmjopen-2019-032701 [Full text]
Although the International Committee of Medical Journal Editors (ICMJE) recommendations for trial data sharing have been controversial, little is known about the attitudes and experiences of authors published in journals with existing data sharing policies. In this collaborative study with Yale University School of Medicine we conducted a self-administered online survey of the authors of clinical trials published January 1, 2012, through March 1, 2016, in 3 high-impact journals with policies either requiring all clinical trial authors to share data (PLOS Medicine) or publish a statement specifying whether they were willing to share data (The BMJ and Annals of Internal Medicine). The survey addressed sharing plans, receipt of sharing requests, and effort required to respond to sharing requests. Respondents were also asked about willingness to share data in 6 hypothetical scenarios.
Tannenbaum S, Ross J, Krumholz, H, Desai N, Ritchie J, Lehman R, Gamble G, Bachand J, Schroter S, Groves T, Gross C. Early experiences with data sharing policies: A survey of published clinical trial investigators. Ann Intern Med. [Epub ahead of print 17 July 2018] doi: 10.7326/M18-0723.
Analysis of trial documentation has revealed that some industry-funded trials may be done more for marketing purposes than scientific endeavour. In this study we aimed to define characteristics of drug trials that appear to be influenced by marketing considerations and estimate their prevalence. We reached consensus that a fifth of drug trials published in the highest impact general medical journals in 2011 had features that were suggestive of being designed for marketing purposes. Each of the marketing trials appeared to have a unique combination of features reported in the journal publications.
Barbour V, Burch D, Godlee F, Heneghan C, Lehman R, Perera R, Ross JS, Schroter S. Characterisation of trials where marketing purposes have been influential in study design: a descriptive study. Trials 2016;17:31 (doi:10.1186/s13063-015-1107-1). [Full text]
Adding, omitting or changing prespecified outcomes can result in bias because it increases the potential for unacknowledged or post hoc revisions of the planned analyses. Journals have adopted initiatives such as requiring the prospective registration of trials and the submission of study protocols to promote the transparency of reporting in clinical trials. The main objective of this feasibility study was to document the frequency and types of outcome discrepancy between prespecified outcomes in the protocol and reported outcomes in trials submitted to The BMJ.
Weston J, Dwan K, Altman D, Clarke M, Gamble C, Schroter S, Williamson PR, Kirkham JJ. A feasibility study to examine discrepancy rates in pre-specified and reported outcomes in articles submitted to The BMJ. BMJ Open 2016;6:e010075. [Full text]
We evaluated the characteristics and publication fate of improperly registered clinical trials submitted to The BMJ over a 4-year period to identify common types of registration issues and their relation to publication outcomes. 123 research papers reporting apparently unregistered or retrospectively registered clinical trials were identified. 110 studies (89.4%) were retrospectively registered, nine (7.3%) were unregistered, three (2.4%) had been registered in an unapproved registry and one study originally lacking registration details was later discovered to have been prospectively registered. Of those papers submitted to The BMJ through the end of 2015, 67 of the 70 papers rejected for registration problems (96%) were subsequently published in another journal. The registration problem was disclosed in only 2 (3%). Improper registration remains a problem. Nonetheless, improperly registered trials are almost always published, suggesting that medical journal editors may not actively enforce registration requirements.
Loder E, Loder S, Cook S. Characteristics and publication fate of unregistered and retrospectively registered clinical trials submitted to The BMJ over 4 years.BMJ Open 2018;8:e020037. doi: 10.1136/bmjopen-2017-020037.[Full text]
Researchers’ awareness and use of authorship guidelines: an international survey
Sara Schroter, Ilaria Montagni, Elizabeth Loder, Matthias Eikermann, Elke Schaeffner, Tobias Kurth
We conducted a large international survey on perceptions and practice of biomedical authorship with active researchers with a wide range of research interests and experience. We sampled active researchers submitting papers to biomedical journals from a range of specialties and Impact Factors to determine: their familiarity with and use of ICMJE criteria; institutional encouragement to use authorship guidelines; how frequently they had experienced inappropriate authorship (honorary and ghost authorship); the timing of authorship decisions in relation to the last paper they coauthored; perceived fairness of authorship decisions; and awareness of their own institution’s authorship policy.
Current status: Paper in preparation. Abstract presented at the Eighth International Congress on Peer Review and Biomedical Publication, Chicago, USA, September 2017. [Abstract]
Observational study of gender bias in editorial decision making
Angèle Gayet-Ageron, Angela Huttner, Alexandra Calmy, Leonard Leibovici, Sara Schroter, Marco De Ambrogi, Peter Donnelly
Several BMJ journals are participating in this collaborative study led by Angèle Gayet-Ageron at the University Hospitals of Geneva. The study will evaluate the influence of first authors’ gender on manuscript acceptance for original research articles sent for peer review.
Current status: Design stage.
Peer reviewed evaluation of registered end-points of randomised trials (the PRE-REPORT study): a cluster-randomised trial
Christopher Jones, Amanda Adams, Mark Weaver, Sara Schroter, Benjamin Misemer, David Schriger, Timothy Platts-Mills
The persistently high rate of selective outcome reporting among published clinical trials indicates that the current peer review process at most journals does not effectively address the problem of selective outcome reporting. Several BMJ journals are participating in this collaborative trial, led by Christopher Jones at Cooper Medical School of Rowan University, to test whether providing peer reviewers with a summary of registered, prespecified primary trial outcomes decreases inconsistencies between prospectively registered and published primary outcomes.
Current status: Protocol published. Jones CW, Adams A, Weaver MA, et al. Peer reviewed evaluation of registered endpoints of randomised trials (the PRE-REPORT study): protocol for a stepped-wedge, cluster randomised trial. BMJ Open 2019;9:e028694. doi:10.1136/ bmjopen-2018-028694. [PDF]
Impact of a short version of the CONSORT checklist for peer reviewers to improve the reporting of randomised controlled trials: a randomised controlled trial
Speich B, Schroter S, Briel M, Moher D, Puebla I, Clark A, Schlussel MM, Ravaud P, Boutron I, Hopewell S
Transparent and accurate reporting is essential for readers to adequately interpret the results of a study. Journals can play a vital role in improving the reporting of published randomised controlled trials. Several BMJ Journals are taking part in a collaborative randomised controlled trial, led by Benjamin Speich at Oxford University, to evaluate the effect of asking peer reviewers to check whether the most important and poorly reported CONSORT (CONsolidated Standards for Reporting Trials) items are adequately reported.
Current status: In progress. [Protocol]
Psychometric testing of ARCADIA, a tool for assessing peer review report quality in biomedical research
Superchi C, Glonti K, Schroter S, Sànchez Espigares JA, Recchioni A, Hren D, Boutron I, González JA
We are evaluating the psychometric properties of the newly developed ARCADIA tool. ARCADIA is the first tool that has been systematically developed to assess the quality of peer review reports in biomedical research. This study is being conducted by Cecilia Superchi in collaboration with BMJ Publishing Group as part of the MIROR doctoral training programme funded by Marie Skłodowska-Curie Actions, dedicated to Methods in Research on Research (MIROR).
Current status: Data analysis and manuscript draft. Abstract accepted for presentation at the PEERE conference on peer review, Valencia, Spain, March 2020. [PDF]
Submitted but unpublished trials: an empirical evaluation
John Ioannidis, Sara Schroter, Doug Altman
There is considerable evidence that some randomised controlled trials (RCTs) are never published or are published with great delay. Publication bias is considered to depend mostly on the fact that investigators may never submit their RCT for publication, and this may be more likely for trials with “negative” results. However, there is more debate and less evidence on whether RCTs may still remain unpublished even though their investigators make an attempt to publish them and do submit them for peer-review and publication. To answer this question, we are studying a cohort of RCTs that were submitted for publication to The BMJ between 1998 and 2001 and examining whether any of them remain unpublished after a long period of time.
Current status: Abstract accepted for presentation at the Sixth International Congress on Peer Review and Biomedical Publication, 2009.
Evaluating an editorial intervention to reduce spin in the abstract conclusion of manuscripts: a randomised controlled trial
Mona Ghannad, Bada Yang, Mariska Leeflang, Adrian Aldcroft, Patrick Bossuyt, Sara Schroter, Isabelle Boutron
We are conducting a two-arm parallel-group randomised controlled trial to obtain an initial estimate of the effectiveness of an editorial intervention to reduce interpretation bias or 'spin' in the abstract conclusions of primary research and research synthesis manuscripts. Research manuscripts sent for peer review will be randomly allocated to the intervention or control group. In the control group, the handling editors will send recommended revisions and reviewers’ comments to authors in their usual manner. In the intervention group, the authors will receive additional instructions alongside the peer reviewers’ comments inviting them to check for and remove spin in the abstract of their revised manuscript. This study is being conducted by Mona Ghannad in collaboration with BMJ Publishing Group as part of the MIROR doctoral training programme funded by Marie Skłodowska-Curie Actions, dedicated to Methods in Research on Research (MIROR).
Current status: Data analysis and manuscript draft. Abstract accepted for presentation at the REWARD EQUATOR conference, Berlin, Germany, February 2020. [PDF] and the PEERE conference on peer review, Valencia, Spain, March 2020 [PDF]
• Schroter S, Tite L., Kassem A. Financial support at the time of paper acceptance: a survey of three medical journals. Learned Publishing 2006;19:291-7. [Abstract]
• van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA 1998;280:234-7 [Full text]
• van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol 1999;52:625-29 [Abstract]
• van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: A randomised trial. BMJ 1999;318:23 - 27 [Abstract] [Full text]
• Schroter S, Black N, Evans S, Smith R, Carpenter J, Godlee F. Effects of training on quality of peer review: a randomised controlled trial. BMJ 2004;328:673-5 [Full text] [Peer review training pack]
• Schroter S, Tite L, Hutchings A, Black N. Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. JAMA 2006;295:314-7. [Full text]
• White I, Carpenter J, Evans S, Schroter S. Eliciting and using expert opinions about dropout bias in randomised controlled trials. Clin Trials 2007;4(2):125-39. [Abstract]
• Tite L, Schroter S. Why do reviewers decline to review for journals? A survey. J Epidemiol Community Health 2007;61:9-12. [Abstract]
• Schroter S, Black N, Evans S, Godlee F, Osorio L, Smith R. What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med 2008:101:507-14. [Full text PDF]
• van Rooyen S, Delamothe T, Evans SJW. Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ 2010;341:c5729 [Full text ]
• Schroter S, Groves T, Højgaard L. Surveys of current status in biomedical science grant review: funding organisations’ and grant reviewers’ perspectives. BMC Med 2010;8:62. [Full text PDF]
• Jones CW, Adams A, Weaver MA, et al. Peer reviewed evaluation of registered endpoints of randomised trials (the PRE-REPORT study): protocol for a stepped-wedge, cluster randomised trial.BMJ Open 2019;9:e028694. doi:10.1136/ bmjopen-2018-028694 [Full text]
• Barnett A, Mewburn I, Schroter S. Working 9 to 5, not the way to make an academic living: observational analysis of manuscript and peer review submissions over time. BMJ 2019; 367 :l6460. [Full text]
• Pace NA, Owen M, Schroter S. What happens to papers rejected by the BMJ on ethical grounds. BMJ 2008;337:a1880. [Letter]
• Dixon-woods M, Foy C, Hayden C, Al-Shahi Salman R, Tebbutt S, Schroter S. Can an ethics officer role reduce delays in research ethics approval? A mixed-method evaluation of an improvement project. BMJ Open 2016;6:e011973.[Full text]
• Schroter S, Morris J, Chaudhry S, Smith, R, Barratt H. Does the type of competing interest statement affect reader perceptions of the credibility of research? A randomised trial. BMJ 2004;328:742-3. [Full text]
• Haivas I, Schroter S, Waechter F, Smith R. Editors' declaration of their own conflicts of interest. CMAJ 2004;171:475-6 [Full text]
• Schroter S, Pakpoor J, Morris J, Chew M, Godlee F. The effect of different financial competing interest statements on readers’ perceptions of clinical educational articles: Study protocol for a randomised controlled trial. BMJ Open 2016;6:e012677. [Full text]
• Schroter S, Pakpoor J, Morris J, Chew M, Godlee F. Effect of different financial competing interest statements on readers’ perceptions of clinical educational articles: A randomised controlled trial. BMJ Open 2019;9:e025029. [Full text]
• Barratt H, Schroter S, Smith R. Two postal surveys of different methods of communicating rejection to authors submitting to a general medical journal. Accountability in Research: Policies and Quality Assurance 2003;10:289-99.
• Schroter S, Jenkins D, Playle R, Walsh K, Probert C, Kellner T, Arnhofer G, Owens D. Evaluation of an online diabetes needs assessment tool (DNAT) for health professionals: a randomised controlled trial. Trials 2009;10:63. [Full text]
• Schroter S, Jenkins RD, Playle RA, Walsh KM, Probert C, Kellner T, et al. Evaluation of an online interactive diabetes needs assessment tool (DNAT) versus online self-directed learning: a randomised controlled trial. BMC Med Educ 2011;11:35. [Full text]
• Murray S, Lazure P, Schroter S, Leuschner PJ, Posel P, Kellner T, et al. International challenges without borders: a descriptive study of family physicians’ educational needs in the field of diabetes. BMC Fam Pract 2011;12:27. [Full text]
• Müllner M, Waechter F, Schroter S, Squire B. How should abridged scientific articles be presented in journals? A survey of readers and authors. CMAJ 2005;172:203-5 [Full text]
• Schriger DL, Sinha R, Schroter S, Liu PY, Altman DG. From submission to publication: a retrospective review of the tables and figures in a cohort of randomized controlled trials submitted to the British Medical Journal. Ann Emerg Med 2006;48:750-6. [Abstract]
• Schroter S, Glasziou P, Heneghan C. Quality of descriptions of treatments: A review of published randomised controlled trials. BMJ Open 2012;2:e001978. doi:10.1136/bmjopen-2012-001978. [Full text]
• Schroter S, Price A, Clarke M. The role of supplementary material in journal articles: Surveys of authors, reviewers, and readers. BMJ Open 2018;8:e021753. doi: 10.1136/bmjopen-2018-021753.[Full text]
• Schroter S, Price A, Flemyng E, et al. Perspectives on involvement in the peer-review process: surveys of patient and public reviewers at two journals. BMJ Open 2018;8:e023357. doi:10.1136/bmjopen-2018-023357. [Full text]
•Schroter S, Price A, Malički M, et al Frequency and format of clinical trial results dissemination to patients: a survey of authors of trials indexed in PubMed. BMJ Open 2019;9:e032701. doi: 10.1136/bmjopen-2019-032701. [Full text]
The BMJ has been at the forefront of innovative thinking and practice in healthcare since 1840 and is the fourth most highly cited general medical journal in the world. Our mission is to lead the debate on health and to engage, inform, and stimulate doctors, researchers, and other health professionals in ways that will improve outcomes for patients. For over 20 years we have been conducting collaborative research into the science of publication including peer review, authorship, conflicts of interest, and publication ethics. We are now partnering with the group meta-research at Maastricht University in the Netherlands to offer a 4 year PhD program in the responsible conduct of publishing scientific research.
The PhD will be embedded within the Care and Public Health Research Institute (CAPHRI) of Maastricht University. CAPHRI is one of Maastricht University’s largest and oldest schools with a track record to high quality, multidisciplinary research in Public Health and Health Care and with a strong focus on having Societal Impact. The institute attracts top scientists from around the world and offers a comprehensive PhD training programme for young scientists.
This new PhD opportunity is for a self-funded or scholarship-based program with distance-learning. PhD candidates will be supervised by a team of research integrity researchers and BMJ staff on selected topics. Successful students will be reimbursed up to a maximum of 10k euro for the 4 years for travel (for one week annual stays at BMJ and Maastricht University), course and conference participation and consumables. Students will also have free access to selected courses. There will be no University tuition fee.
Applicants should select one of the projects below, but we are also open to other good ideas.
1) Open peer review
2) Strengthening peer review of observational studies
3) Handling retractions in biomedicine
4) Preprints and post publication peer review in biomedicine
5) Preregistration and registered reports in biomedicine
6) Replication studies in biomedicine
7) Is peer review effective?
Successful candidates are expected to be highly motivated and hold a Masters Degree in a relevant discipline. They will be asked to provide relevant documentation to show they:
• are fluent in English (IELTS>6.5 or equivalent);
• have the necessary time available to do the research (e.g. recommendation letter from current employer); and
• have proven scientific writing ability (e.g. at least one coauthored journal article).