Starling Group, co-chaired by Trish Groves (The BMJ) and Professor Liselotte Højgaard (European Medical Research Councils)There is now a considerable body of evidence on the methods, outcomes, effectiveness, best practice, and ethics of editorial peer review of submissions to biomedical journals. In contrast, grant review, which is practised in a wide variety of ways, has a relatively sparse evidence base. It is possible that grant review might be informed by editorial peer review practice and evidence. The Starling Group is a newly set up international group interested in improving grant review for health and medical research. The Starling Group conducted two surveys. The first was an international survey of biomedical research funding organisations to measure typical workloads of funding organisations and current problems they face. The second was an international survey of external grant reviewers to measure typical workloads of external grant reviewers and factors that motivate them to review.
Schroter S, Groves T, Højgaard L. Surveys of current status in biomedical science grant review: funding organisations’ and grant reviewers’ perspectives. BMC Medicine 2010;8:62. [Full text PDF]
We conducted an RCT to see whether telling peer reviewers that their signed reviews of original research papers might be posted on The BMJ's website would affect the quality of their reviews.
We found that this had no important effect on review quality. Although the possibility of posting reviews online was associated with a high refusal rate among potential peer reviewers and an increase in the amount of time taken to write a review, we believe that the ethical arguments in favour of open peer review more than outweigh these disadvantages.
van Rooyen S, Delamothe T, Evans SJW. Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ 2010;341:c5729 [Full Text]
We conducted a randomised controlled trial to evaluate the effect on the quality of peer review of blinding reviewers to authors' identities and requiring reviewers to sign their reports. We found that neither blinding reviewers to the authors and origin of the paper nor requiring them to sign their reports had any effect on the rate of detection of errors. Such measures are unlikely to improve the quality of peer review reports.
Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA 1998: 280: 237-240. [Abstract]]
We conducted a randomised controlled trial to determine whether concealing authors' identities from reviewers (blinding) and/or revealing the reviewer's identity to a coreviewer (unmasking) affects the quality of reviews, the time taken to carry out reviews, and the recommendation regarding publication. We found that blinding and unmasking made no editorially significant difference to review quality, reviewers' recommendations, or time taken to review.
van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review: A randomized trial. JAMA 1998; 280: 234 – 7. [Abstract]
We conducted a randomised controlled trial to examine the effect on peer review of asking reviewers to have their identity revealed to the authors of the paper. We found that this had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review.
van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: A randomised trial. BMJ 1999; 318: 23 – 27. [Abstract] [Full Text]
We developed and validated an instrument to measure the quality of reviews. The Review Quality Instrument (RQI) assesses the extent to which a reviewer has commented on five aspects of a manuscript (importance of the research question, originality of the paper, strengths and weaknesses of the method, presentation, interpretation of results) and on two aspects of the review (constructiveness and substantiation of comments). The RQI’s internal consistency was high (Cronbach's alpha 0.84). The mean total score (based on the seven items each scored on a 5-point Likert scale from 1 to 5) had good test-retest (Kw = 1.00) and inter-rater (Kw = 0.83) reliability. There was no evidence of floor or ceiling effects, construct validity was evident, and the respondent burden was acceptable (2-10 minutes). The instrument can be recommended for use in the study of peer review and we have used it in several of our studies.
van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. Journal of Clinical Epidemiology 1999; 52(7): 625 – 629. [Abstract]
We conducted a study to determine the characteristics of reviewers for a general medical journal who produce high-quality reviews and to describe the characteristics of a good review, particularly in terms of the time spent reviewing and turnaround time. In a logistic regression analysis, the only significant factor associated with higher-quality ratings by both editors and authors was reviewers trained in epidemiology or statistics. Younger age also was an independent predictor for editors' quality assessments, while reviews performed by reviewers who were members of an editorial board were rated of poorer quality by authors. Review quality increased with time spent on a review, up to 3 hours but not beyond.
Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for a general medical journal? JAMA 1998; 280(3): 231 – 233. [Abstract]
Many studies have illustrated the inadequacies of peer review and its limitations in improving the quality of research papers. However, few studies have evaluated interventions that try to improve peer review, and there have been no previous randomised controlled trials to examine the effects of training. Training that would be feasible for reviewers to undergo and for a journal to provide would have to be short or provided at a distance. Although the effectiveness of short educational interventions is questionable, some brief interventions have been shown to be successful (depending on what is being taught and the methods used). We aimed to determine whether reviewers for The BMJ who underwent training would produce reviews of better quality than those who received no training; whether face to face training would be more beneficial than a self taught package; and whether any training effect would last at least six months. The training materials for the trial are now freely available on bmj.com.
Schroter S, Groves T. BMJ training for peer reviewers. BMJ 2004; 328. [Full Text]
White I, Carpenter J, Evans S, Schroter S. Eliciting and using expert opinions about dropout bias in randomised controlled trials. Clinical Trials: Journal of the Society for Clinical Trials 2007;4(2):125-39. [Abstract]
Schroter S, Black N, Evans S, Godlee F, Osorio L, Smith R. What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med 2008:101: 507–514. [Full Text]
Many journals give authors the opportunity to suggest reviewers to review their paper, but editors’ use of these reviewers vary as some are concerned that they may favour the author. We conducted a study comparing author- and editor-suggested reviewers of ten biomedical journals in a range of specialties to investigate differences in review quality and recommendation for publication. We found no difference in review quality but author-suggested reviewers tended to make more favourable recommendations for publication.
Schroter S, Tite L, Hutchings, A, Black N. Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. JAMA 2006;295:314-7. [Abstract] [Fulltext pdf]
Peer reviewers are usually unpaid and their efforts are not formally acknowledged. Editors of some journals experience difficulty finding appropriate reviewers who are able to complete timely reviews, resulting in publication delay. We conducted a survey of reviewers from five BMJ Journals to determine why reviewers decline to review and their opinions of various incentives. We found that reviewers are more likely to accept to review a manuscript when it is relevant to their area of interest and lack of time is the principal factor in the decision to decline. Most respondents agreed that financial incentives would not be effective when time constraints are prohibitive
Conflict of interest has been defined as a set of conditions in which professional judgment concerning a primary interest (such as patient welfare or the validity of research) can be influenced by a secondary interest (such as financial gain). Despite increasing evidence that conflict of interest influences authors' conclusions, there has been little research into the effect on readers' perceptions of research. We conducted two studies to see whether a declarations of competing interests influences readers' perceptions of the interest, importance, relevance, validity, and believability of a study.
Chaudhry S, Schroter S, Smith R, Morris J. Does declaration of competing interests affect reader perceptions: A randomised trial? BMJ 2002; 325:1391-2. [Full Text]
Schroter S, Morris J, Chaudhry S, Smith, R, Barratt H. Does the type of competing interest statement affect reader perceptions of the credibility of research? A randomised trial. BMJ 2004; 328:742-3. [Full Text]
Williams HC, Naldi L, Paul C, Vahlquist A, Schroter S, Jobling R. Conflicts of Interest in Dermatology. Acta Derm Venereol. 2006;86(6):485-97. [Abstract]
The International Committee of Medical Journal Editors (ICMJE) recognises that all persons involved in peer review and journal publishing can have conflicting interests. Although authors are expected to declare conflicts of interest at the time of publication, there is no requirement for journal editors to publicly declare their own competing interests. We conducted a survey of editors of General and Internal Medicine journals to explore their policies regarding competing interests of editors and other journal staff.
Haivas I, Schroter S, Waechter F, Smith R. Editors' declaration of their own conflicts of interest. Canadian Medical Association Journal 2004; 171(5): 475-476. [Full Text PDF]
We investigated whether editors at The BMJ could make a decision about manuscripts based on reading only the abstract and how this initial decision differed from when they read the whole submission. Editors were able to make an editorial decision based on reading only the abstract for 62% (229/372) of submissions. Where a decision could be made, there was no significant difference between the two assessments in the proportion of papers rejected immediately by a single editor. Neither was there a significant difference between the two assessments in the proportion of papers where it was suggested that the submission should be sent to an external reviewer.
Since the introduction of the world wide web, peer-reviewed journal publishers have been wrestling with the dilemma of how to survive in a market place that facilitates and encourages the free exchange of electronic content. This web-based movement is in opposition to the traditional subscriptions-based business model of scientific publishing that relies heavily on print-based pricing and distribution. One emerging alternative to the traditional journal business model is Open Access publishing where lawful free access to journal content is provided online with production funded through means other than subscription charges. Various models for funding open access have been proposed, but one in particular has generated considerable interest and has become the business model for several new open access journals. The 'author-pays' model requires that authors, or their granting agencies or institutions, pay a contribution to the cost of publishing on submission and / or acceptance of their paper. We conducted some research with authors to help understand their perspectives. The research comprised several stages:
We conducted semi-structured telephone interviews to explore authors' attitudes towards open access publishing and author charges, their perceptions of journals that charge authors, and whether they would be willing to submit to these journals. We found authors were more aware of the concepts of open access publishing and author pays models than previously reported. Almost all authors supported the concept of open access, but few had submitted to an open access journal, other than The BMJ. Reasons for not submitting included lack of awareness of which journals publish with open access, and journal quality taking a higher priority in decision making than the availability of open access. Authors disliked the idea of author charges without institutional support and were concerned about implications for authors from developing countries and those without research funding.
Schroter S, Tite L, Smith R. Perceptions of Open Access Publishing: Interviews with Journal Authors. BMJ 2005;330:756-9. [Full Text]
We conducted an electronic survey of 468 journal authors to assess current knowledge and perceptions of open access and author-pays publishing. Compared with non-open access subscription-based journals, 35% agreed that open access author-pays journals have a greater capacity to publish more content making it easier to get published, 27% thought they had lower impact factors, 31% thought they had faster and more timely publications, and 46% agreed that people will think anyone can pay to get published. Over half thought they would have to make a contribution or pay the full cost of an author charge.
Schroter S, Tite L. Open access publishing and author-pays business models: A survey of authors’ knowledge and perceptions. J R Soc Med 2006;99:130-7.[Full Text PDF]
The author-pays model (open access publishing funded through author charges) is dependent on authors having access to financial support at the time their research papers are accepted. We conducted a survey of 377 authors from three journals in different specialties to determine the availability of external funding for publication charges at different points in the research process. We found a large proportion of published research is not externally funded, and many funded researchers do not have access to financial support at the time their paper is accepted for publication.
Schroter S, Tite L., Kassem A. Financial support at the time of paper acceptance: A survey of three medical journals. Learned Publishing 2006;19(4):291-7. [Full Text PDF]
We carried out a survey to determine whether free access to research articles on bmj.com is an important factor in authors' decisions on whether to submit to the The BMJ, whether the introduction of access controls to part of the The BMJ's content has influenced authors' perceptions of the journal, and whether the introduction of further access controls would influence authors' perceptions. We found authors value free access to research articles and consider this an important factor in deciding whether to submit to The BMJ. Closing access to research articles would have a negative effect on authors' perceptions of the journal and their likeliness to submit.
Schroter S. Importance of free access to research articles on decision to submit to the BMJ: A survey of authors. BMJ 2006;332:394-6. [Full Text]
We investigated the nature and frequency of statistician involvement in medical research and its relation to the final editorial decision. Authors submitting to The BMJ and Annals of Internal Medicine in 2001 were sent a questionnaire asking if they received assistance from a person with statistical expertise, the nature of any such contribution, and reasons why, if no statistical input was received. We found that statistical input to medical research is widely recommended but inconsistently obtained. Individuals providing such expertise are often not involved until the analysis of data and many go unrecognised by either authorship or acknowledgment.
It is important that authors understand why their work has been rejected by a journal as this may influence the publication, and whether and where they submit in future. Little research has been conducted into authors’ perceptions of editorial review. We conducted two cross-sectional surveys of The BMJ authors whose papers were rejected without external peer review. The purpose was to evaluate differences in perceptions of the review system according to method of rejection and to evaluate preferred methods of receiving rejection.
Barratt H, Schroter S, Smith R. Two postal surveys of different methods of communicating rejection to authors submitting to a general medical journal. Accountability in Research: Policies and Quality Assurance 2003;10(4):289 – 299. [Abstract]
Editorial space is a scarce resource and editors struggle to meet the expectations of both authors and readers. Several scientific journals have adopted strategies to publish detailed article content on the web and a more concise presentation in the print journal. We elicited readers’ and authors’ preferences for three different presentational styles of abridged print versions: the conventional approach where the Introduction-Methods-Results-Discussion structure is maintained and the text simply shortened, a journalistic version with a very different structure, and a version with an enhanced abstract similar to those used in evidence-based medicine journals. We found authors and readers prefer a more conventional style of abridged papers.
We conducted a survey to evaluate how acceptable authors find the The BMJ's current practice of publishing short versions of research articles in the paper journal and a longer version on the web (ELPS) and to determine authors’ attitudes towards publishing only abstracts in the paper journal and publishing unedited versions on bmj.com once papers have been accepted for publication.
Tables and figures can convey details and complex relationships not easily described in text. We characterised the quantity and quality of data tables and figures in reports of randomised controlled trials (RCTs) submitted to The BMJ and published in peer-reviewed journals. We investigated how the peer review process affected table and figure quality. We found the numbers of tables and figures did not change markedly between submission and publication. Less than half the figures met their data presentation potential, with most failing to portray by-subject data and few displaying advanced features such as pairing, symbolic dimensionality, or small multiples. The BMJ external peer reviewers seldom commented on tables or figures.
Schriger DL, Sinha R, Schroter S, Liu PY, Altman DG. From Submission to Publication: A Retrospective Review of the Tables and Figures in a Cohort of Randomized Controlled Trials Submitted to the British Medical Journal. Ann Emerg Med 2006 Dec;48(6):750-6. [Abstract]
Authors are required to describe in their manuscripts ethical approval from an appropriate committee and how consent was obtained from subjects when research involves human subjects. Previous studies have focused on concordance with these regulations within a single speciality or for clinical trials. We assessed reporting of these protections for several study designs published in Annals of Internal Medicine, The BMJ, JAMA, Lancet and New England Journal of Medicine. We found the reporting of ethical approval and consent has improved for RCTs but journals are less good at reporting this information for other study designs.
Schroter S, Plowman R, Hutchings A, Gonzalez A. Reporting of ethical committee approval and patient consent by study design in five general medical journals. J Med Ethics 2006;32:718-23. [Abstract] [Full text PDF]
The BMJ has also been a study site for external research projects. Here are two examples of such projects:
Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, Bero LA, University of California, San Francisco.
Publication bias has been shown to exist, but little is known about the role that journal editors and reviewers play in publication bias. The goal of the study was to explain systematic biases in the editorial decision-making process by identifying factors that influence editors' decisions to accept or reject articles for publication in biomedical journals. UCSF researchers examined editorial practices, processes, and outcomes at three major biomedical journals in the UK (The BMJ and the Lancet) and the US (Annals of Internal Medicine. They concluded that submitted manuscripts are more likely to be published if they have high methodological quality, RCT study design, descriptive or qualitative analytical methods and disclosure of any funding source, and if the corresponding author lives in the same country as that of the publishing journal.
Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, Bero LA. Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust. 2006 Jun 19;184(12):621-6. [Abstract]
BMJ Knowledge in collaboration with Centre for Health Sciences Research, Cardiff University
Informed medical decision-making depends in part on an understanding of the benefits and harms of different treatment options. There is evidence that both patients and physicians have difficulty interpreting, incorporating, and remembering information on risk in clinical practice. This is particularly so when considering the statistical aspects of risk and how these are communicated between patients and physicians. There are various challenges to effective and appropriate risk communication. Bias may be introduced by the way risk information is presented. This randomised controlled trial explored ways to present quantitative and qualitative information on medical risk in a format more accessible to patients. Formats included general descriptions, bar graphs and comparisons to everyday risks such as getting struck by lightening. The study assessed the efficacy of these risk formats in improving outcomes related to patient decision-making.
Edwards A, Thomas R, Williams R, Ellner AL, Brown P, Elwyn G. Presenting risk information to people with diabetes: evaluating effects and preferences for different formats by a web-based randomised controlled trial. Patient Educ Couns. 2006 Nov;63(3):336-49[Abstract].