Uptake of systematic reviews and meta-analyses based on individual participant data in clinical practice guidelines: descriptive studyBMJ 2015; 350 doi: http://dx.doi.org/10.1136/bmj.h1088 (Published 06 March 2015) Cite this as: BMJ 2015;350:h1088
- Claire L Vale, senior research scientist1,
- Larysa H M Rydzewska, research scientist1,
- Maroeska M Rovers, professor of evidence based surgery2,
- Jonathan R Emberson, senior statistician3,
- François Gueyffier, professor of clinical pharmacology45,
- Lesley A Stewart, director6
- on behalf of the Cochrane IPD Meta-analysis Methods Group
- 1MRC Clinical Trials Unit at UCL, London WC2B 6NH, UK
- 2Radboud Institute for Health Sciences, Radboud University Medical Centre, Nijmegen, Netherlands
- 3Clinical Trial Service Unit and Epidemiological Studies Unit (CTSU), University of Oxford, Oxford, UK
- 4Université Claude Bernard Lyon 1, Université de Lyon, Lyon, France
- 5Hospices Civils de Lyon, Service de Pharmacologie Clinique & Essais Thérapeutiques, Bron cedex, France
- 6Centre for Reviews and Dissemination, A/B Block Alcuin College, University of York, York, UK
- Correspondence to: L H M Rydzewska
- Accepted 26 January 2015
Objective To establish the extent to which systematic reviews and meta-analyses of individual participant data (IPD) are being used to inform the recommendations included in published clinical guidelines.
Design Descriptive study.
Setting Database maintained by the Cochrane IPD Meta-analysis Methods Group, supplemented by records of published IPD meta-analyses held in a separate database.
Population A test sample of systematic reviews of randomised controlled trials that included a meta-analysis of IPD, and a separate sample of clinical guidelines, matched to the IPD meta-analyses according to medical condition, interventions, populations, and dates of publication.
Data extraction Descriptive information on each guideline was extracted along with evidence showing use or critical appraisal, or both, of the IPD meta-analysis within the guideline; recommendations based directly on its findings and the use of other systematic reviews in the guideline.
Results Based on 33 IPD meta-analyses and 177 eligible, matched clinical guidelines there was evidence that IPD meta-analyses were being under-utilised. Only 66 guidelines (37%) cited a matched IPD meta-analysis. Around a third of these (n=22, 34%) had critically appraised the IPD meta-analysis. Recommendations based directly on the matched IPD meta-analyses were identified for only 18 of the 66 guidelines (27%). For the guidelines that did not cite a matched IPD meta-analysis (n=111, 63%), search dates had preceded the publication of the IPD meta-analysis in 23 cases (21%); however, for the remainder, there was no obvious reasons why the IPD meta-analysis had not been cited.
Conclusions Our results indicate that systematic reviews and meta-analyses based on IPD are being under-utilised. Guideline developers should routinely seek good quality and up to date IPD meta-analyses to inform guidelines. Increased use of IPD meta-analyses could lead to improved guidelines ensuring that routine patient care is based on the most reliable evidence available.
Systematic reviews have enormous potential value in developing healthcare policy and clinical guidelines, as well as directly providing clear and reliable evidence to those making clinical decisions. Individual participant data (IPD) meta-analyses are commonly described as the gold standard of systematic reviews. Unlike systematic reviews and meta-analyses based on aggregate data (which is usually extracted from study reports) the results of IPD meta-analyses are not so restricted by many of the well documented publication or reporting biases that can hamper other non-IPD systematic reviews.1 2 3 4 5 They often include greater numbers of patients from more trials, both published and unpublished, providing more reliable summaries of trial results. Furthermore, using comprehensive IPD can generate more detailed results. For example, the data can be better used to test whether the effect of a given treatment differs in one type of patients compared with another (treatment-covariate interactions), thus potentially enabling better targeting of treatments within specific patient populations. Detailed information on the exact timing of particular outcomes also allows for a proper assessment of how any treatment benefits (or hazards) might evolve with duration of treatment (treatment-time interactions). Consequently, systematic reviews and meta-analyses of IPD can provide key evidence to inform and influence clinical guidance.
Yet despite this clear potential, the extent to which IPD meta-analyses, are actually used in practice by guideline developers, as best or level 1 evidence,6 is uncertain. Therefore, following an international workshop of 31 people drawn from the members of the Cochrane IPD Meta-analysis Methods Group (September 2012, London), we assessed whether and how such meta-analyses are used in published clinical guidelines.
We identified where practice recommendations had been based on evidence from IPD meta-analyses and ascertained whether findings from such meta-analyses were given precedence over systematic reviews using aggregated data and dealing with the same question, given the known advantages of using IPD.7 8 9 Our ultimate intention was to help inform those conducting IPD meta-analyses and those developing guidance, on how to optimise the uptake and use of these meta-analyses.
Study sample identification
Systematic reviews and meta-analyses of IPD—We extracted these from a database maintained by the Cochrane IPD Meta-analysis Methods Group,10 supplemented by records of published IPD meta-analyses held in a separate database.11 To be included in our study, the meta-analyses should have been published either in the Cochrane Library12 or in a peer reviewed journal between 1 January 2008 and 31 December 2010 and carried out a meta-analysis of IPD as part of a full systematic review of randomised controlled trials. They should have evaluated at least one treatment intervention, irrespective of healthcare area or geographical region.
Clinical guidelines—We searched the US National Guideline Clearinghouse;13 the Scottish Intercollegiate Guidelines Network,14 and the UK National Institute for Health and Care Excellence15 websites for all current clinical guidelines in healthcare areas relating to our sample of IPD meta-analyses. For guidelines related to cancer treatments, we also searched the Standards and Guidelines Evidence directory of cancer guidelines.16 Eligible guidelines should have been published either originally or in an updated form between 1 January 2010 and 31 December 2013, a timeframe that was chosen to allow for the inclusion of IPD meta-analyses from our sample; albeit that we accepted a one year overlap between publication of guidelines and meta-analyses in 2010. This was done primarily to identify references to the IPD meta-analyses published in 2008-09. Two authors (LHMR and CLV) matched all potentially eligible guidelines to the IPD meta-analyses included in the sample. Guidelines were matched to the meta-analyses if they dealt with the same medical condition, treatment type, and patient populations (for example, adults or children). We also included guidelines with a broad scope that encompassed the subject area of the meta-analysis. Matched guidelines were necessarily published after the publication date of the meta-analysis. Duplicate guidelines (that is, those identified from more than one source) were removed. Two authors (LHMR and CLV) resolved queries on the suitability of individual guidelines for inclusion in the sample by discussion.
For each matched clinical guideline, we extracted information on the guideline developer, date of publication, and free availability of the guideline. We also carried out a thorough text search of each guideline to identify any reference to the matched IPD meta-analyses, using search terms including the authors’ names, name of intervention, systematic review, meta-analysis, individual patient or participant data, and IPD. Furthermore, we sought data on whether any other IPD meta-analysis or meta-analysis of aggregate data was cited, and any key recommendations within the guideline that were attributed to a matched IPD meta-analysis. Two authors (LHMR and CLV) resolved any discrepancies by discussion.
For each meta-analysis in the test sample, we recorded the total number and proportion of unique, matched guidelines that cited it. For each cited meta-analysis, we looked for evidence within the guideline that it had been critically appraised and whether specific practice recommendations had been based on the findings of the meta-analysis. We also noted whether any other related IPD meta-analyses or meta-analyses based on aggregate data had been cited, and where both types of meta-analyses were cited we looked for evidence that any distinction had been made between the two. Finally, for guidelines that did not cite the matched IPD meta-analysis, we looked for any likely reasons why it had not been included or for information about alternative evidence sources that had been used to inform the guideline recommendations.
To obtain overall totals, averages, and frequencies on which interpretation could be based we collated data from each of the IPD meta-analyses across our test sample, including the frequency of citations and the number of guidelines that based their recommendations on results of a matched IPD meta-analysis.
Our two chosen data sources yielded 33 eligible IPD meta-analyses published between 2008 and 2010 (fig 1⇓ and table⇓). Fourteen were reviews of cancer treatments and eight were reviews of interventions related to cardiovascular disorders, mainly stroke and heart disease. The remaining 11 reviews covered a broad range of healthcare areas, including epilepsy, gynaecological disorders, otitis media, and asthma.
Quality of included meta-analyses
Each IPD meta-analysis was conducted as part of a full systematic review. In general, across the 33 meta-analyses, the systematic searches had looked across numerous sources (bibliographic databases, Cochrane Library, conference proceedings, and reference lists). Many (21/33) had also searched trials registers or consulted with relevant experts to identify unpublished or ongoing trials, of which 13 had identified and included data from one or more unpublished trials. Most of the meta-analyses (26/33) reported either having conducted a quality assessment of the included trials, either using a recognised tool (for example, Cochrane risk of bias tool)17 or thoroughly checking and verifying the validity of the data received. Where reported, the IPD meta-analyses were able to include most of the trials identified (median 86%, range 56-100%) and also of the known randomised patients (median 92%, range 65-100%). Methods of analyses used were generally well described and in keeping with the outcomes being analysed, and many had prespecified and reported patient level subgroup analyses in addition to the primary and secondary outcomes.
Guideline sample selection
Our search of clinical guidelines published between 2010 and 2013 yielded 1171 records (fig 2⇓); however, many of these (n=432, 37%) were identified as out of scope based on their titles or guideline summaries, or both (for example, unrelated disease or treatment, or both, or outside the date range) and were excluded. Duplicate records obtained from more than one guideline source, (n=201, 17%) were also discarded. The remaining 538 potentially eligible guidelines were examined in detail. Of these, a further 308 were either found to be ineligible (fig 2) or were excluded for other reasons (n=53)—for example, because a new guideline had superseded the original (n=18) or because the guideline was not freely available (n=7). Therefore, results are based on 177 guidelines published between 2010 and 2013 in clinical areas and populations matching those in the sample of IPD meta-analysis.
The 177 guidelines had been written by 43 (see supplementary appendix 1) different guideline development groups (range 1-8 per developer) and spanned most geographical areas, albeit half (n=89, 50%) were North American guidelines. A further 36% (n=64) were from the United Kingdom and Europe, with the remainder being largely from Australia, New Zealand, and Asia. A matched IPD meta-analysis was cited in 66 of the 177 guidelines (37%). A third of these guidelines (n=22, 34%) had included some critical appraisal of the matched meta-analysis. The box and supplementary table 1 provide examples of guideline recommendations that could be directly attributed to the findings of cited IPD meta-analyses.
Examples of guideline recommendations linked to findings of cited IPD meta-analyses
Example 1*: concomitant chemoradiation for cervical cancer22
Potential impact—making recommendations about specific treatment regimens
The IPD meta-analysis
15 randomised trials, 3452 patients
6% survival benefit at five years associated with chemoradiation
No suggestion of a difference between the treatment benefit, whether platinum based or non-platinum based treatment regimens
Recommendations in guidelines citing IPD meta-analysis
Concomitant chemoradiation represents the standard
Non-platinum based regimens for chemoradiation seem to be as efficient as platinum based chemotherapy
The role of adjuvant chemotherapy after concomitant chemoradiation remains unclear and should be included in further clinical investigations
Versus recommendations in guideline not citing IPD meta-analysis
In women with late cervical cancer: combined chemotherapy and radiotherapy is the treatment of choice.
Example 2†: antibiotics for acute otitis media54
Impact—targeting treatment to patient groups most likely to benefit
The IPD meta-analysis
Six randomised trials, 1643 children aged from 6 months to 12 years
In children younger than 2 years with bilateral acute otitis media, 55% of controls and 30% receiving antibiotics still had pain, fever, or both at 3-7 days, with a rate difference between these groups of −25% (95% confidence interval −36% to −14%)
In children with otorrhoea the rate difference was −36% (95% confidence interval −53% to −19%)
Impact on guidelines
The UK National Institute for Health and Care Excellence (NICE) guideline on the treatment of otitis media recommended that, in general, antibiotics should not be prescribed, or prescription should be delayed. However, immediate antibiotic prescription was recommended for children between the ages of 6 months and 2 years with bilateral acute otitis media, and for children of any age with a diagnosis of otorrhoea and acute otitis media. Previous systematic reviews identified by the guideline developers, based on summary data, had not shown any differences between patient subgroups.
Example 3†: monotherapy for epilepsy55
Impact—providing evidence on appropriate outcome measures
The IPD meta-analysis
20 randomised trials, 6418 patients with epilepsy treated with monotherapy
Network meta-analysis comparing eight antiepileptic drugs
Primary outcomes were time to treatment failure and time to 12 month remission from seizures; secondary outcome was time to first seizure
For partial onset seizures (n=4628 (72%) patients), lamotrigine, carbamazepine, and oxcarbazepine provide the best combination of seizure control and treatment failure
For generalised onset tonic clonic seizures (n=1790 (28%) patients) estimates suggest valproate or phenytoin may provide the best combination of seizure control and treatment failure
Impact on guidelines
Without results from the IPD meta-analysis, the guideline produced by NICE would have been restricted to outcomes reported in the individual trials—for example, proportion of seizure-free participants. In contrast, the IPD meta-analysis provided robust evidence on more informative outcomes—for example, time to treatment failure, time to 12 month remission from seizures, which were used to inform the guideline recommendation
*Information missed when relevant IPD meta-analyses are omitted from guidelines.
†Information gained when relevant IPD meta-analyses are assimilated in guidelines.
Of the 177 guidelines, 69 (39%) cited a systematic review and meta-analysis of aggregate data either alongside the matched IPD meta-analysis (n=29, 42%) or alone. Where both were cited, few guidelines (n=4, 14%) made any distinction between the two systematic reviews—for example, by explicitly stating that one was based on individual patient data. There was no evidence that IPD meta-analyses were more or less likely to be included in guidelines from particular geographical locations. We saw no clear evidence that the uptake of individual IPD meta-analyses depended on the length of time since publication. However, the time to uptake in a specific guideline will vary according to the frequency with which the guidelines are produced or updated. In addition, certain guideline developers seemed more likely than others to cite IPD meta-analyses. More than half of the citations of matched IPD meta-analyses (35/66, 53%) were included in guidelines produced by only seven of the 43 guideline developers in the sample (see supplementary figure 1), whereas 13/43 guideline development groups (responsible for 31/177 guidelines) did not cite any of the eligible meta-analyses.
Matched guidelines were identified for 29 of the 33 meta-analyses included in the sample, with a median of five guidelines for each meta-analysis. The highest number of matched guidelines was for one meta-analysis in non-small cell lung cancer, for which 16 matched guidelines were identified. Of the 29 meta-analyses with matched guidelines, 21 (72%) had been cited in at least one guideline (median 3, range 1-14 guidelines) and eight (28%) were not cited in any of the matched guidelines. Of the 21 meta-analyses that were cited, only five were mentioned in more than half of the relevant guidelines (table 1).
Importantly, around two thirds of the guidelines in our sample (n=111, 63%) did not cite any IPD meta-analysis, despite one being available in every case. Three of these guidelines (2%) had referenced another guideline as their primary evidence source, and on further investigation it was clear that the guideline referenced had used an IPD meta-analysis. For 23 guidelines (21%) we found that the search dates had preceded the publication date of the IPD meta-analysis. A further 11 guidelines (10%) did not supply their evidence sources in a reference list and so we were unable to tell what had led to their recommendations. However, for the rest (n=74, 67%) we could find no obvious reasons why the IPD meta-analysis had not been cited. Although we did not carry out any formal quality appraisal for the included guidelines, we did seek information from either the individual guideline developers’ handbooks or their websites as to whether they advocated the use of the appraisal of guidelines for research and evaluation (AGREE) tool51 52 for guideline development or the grading of recommendations assessment, development, and evaluation (GRADE)53 system for appraising the quality of the included evidence. Of the 43 guideline development groups, we found that 14 guideline developers explicitly promoted the use of either the AGREE or the GRADE tools in their guidance documents and six referred to other methods of appraisal. However, for the remainder there was either no mention of the standards adhered to in the information available to us (n=5) or no information could be found (n=18).
Overall, based on this sample of 33 IPD meta-analyses and 177 matched guidelines, our results indicate that systematic reviews, based on IPD, are being under-utilised as a source of the best available evidence within clinical guidelines. Little more than a third of the guidelines in our sample had cited a recent and relevant IPD meta-analysis, and less than 20% of the guidelines had clearly used information from the meta-analysis in formulating their recommendations. Where such meta-analyses had been cited as part of the evidence base, there was often little or no distinction between an IPD meta-analysis and other non-IPD (or non-systematic) reviews or individual randomised controlled trials, which were often all assigned the highest level of evidence (for example, 1++ or 1A). It was also apparent that certain guideline developers were more likely to use IPD meta-analyses as the basis of their recommendations, as more than half of all the guidelines in our sample that cited a meta-analysis were produced by only seven guideline development groups. It is also unclear why a high proportion of guidelines utilised evidence from existing systematic reviews of aggregate data (or carried out their own analyses based on aggregate data) when relevant systematic reviews and meta-analyses of IPD were available, as was the case for each of the guidelines included in this study. We believe that a relevant IPD meta-analysis should always contribute to clinical guidelines if available, unless it is determined to be of poor methodological quality, based on limited data or superseded by a more recent aggregate data meta-analysis that includes several new trials. None of these exceptions occurred in our sample. Furthermore, even when a more recent aggregate data review is available, pre-existing IPD meta-analyses may still provide important additional information—for example, relating to the effects of an intervention in different subgroups of patients, thus making an important contribution to the overall evidence base. Failure to assimilate information from IPD meta-analyses may lead to less appropriate or limited recommendations being made, whereas their uptake may better inform the guideline recommendations (see box).
Strengths and weaknesses of this study
Ours was not designed to be a comprehensive study but rather to provide insight into the uptake of IPD meta-analyses by guideline developers. Therefore we are aware of certain limitations. Firstly, we restricted our searches to three main sources (four for cancer guidelines) and so may not have identified all relevant guidelines. That many of the IPD meta-analyses identified were of treatment interventions for cancer (14/33), reflects the tradition of IPD meta-analysis in cancer research. We also searched an additional source of cancer specific guidelines. While we accept that the sample may not be entirely representative, we screened more than 1100 guidelines in total and included guidelines produced by several high profile organisations. Secondly, the interval between publication of an IPD meta-analysis and its subsequent uptake in a guideline can vary considerably across guideline development groups. Intervals depend on updating schedules or triggers and on available resources, and usually range from six months to two years. Therefore, although we considered the use of a maximum three year time lag between publication of the meta-analysis (2008-10) and publication of the matched guideline (2010-13) as an appropriate interval, this does not guarantee that we identified all relevant citations. However, we found that for the meta-analyses within this sample, many were incorporated in guidelines within a year of publication. Thirdly, a lack of consistency in the reporting or presentation of guidelines made it difficult to consistently attribute recommendations to the evidence supporting them. For example, some guidelines omitted details of their methods for searching the literature, or provided sparse (or no) reference lists. Therefore it was not always clear if a relevant meta-analysis had not been identified in the searches carried out or whether it would have been identified but not included. This may reflect fundamental differences in the quality of the included guidelines. Although we did not carry out a formal quality assessment of all of the included guidelines, we found that of the seven guideline development groups with guidelines that included more than half of the citations of IPD meta-analyses that we identified, five indicated that at an organisational level they advocated use of either the AGREE tool51 52 (Alberta Health Services, American College of Cardiology Foundation/American Heart Association, American College of Chest Physicians, and Scottish Intercollegiate Guideline Network, SIGN) or the Agency for Healthcare Research and Quality guidelines56 57 (American College of Radiology). Full reporting of evidence sources and methodology used may better enable users to appraise the quality of the guideline.58 Finally, a small number of potentially eligible guidelines were not freely available. While we were able to obtain some of these through our institution’s library, others were not accessible and could not be included. None the less, we believe that the data underpinning our review are robust enough to enable us to draw qualitatively reliable conclusions about the use of IPD meta-analyses in guidelines.
Relation to other studies
To our knowledge this is the first review focussing specifically on the inclusion of IPD meta-analyses in clinical guidelines. With recent calls to improve and increase access to trial data, the number of available such meta-analyses may increase dramatically. Therefore we considered that this study provides an important insight into the current situation on the assimilation of IPD meta-analyses into clinical guidelines. However, others have looked at uptake of systematic reviews in guidelines more generally. For example, recent results from ongoing work at the UK Cochrane Centre have shown that Cochrane reviews across the breadth of healthcare areas covered by the collaboration have been used to inform findings on the background of clinical guidelines produced by NICE, SIGN, and the World Health Organization. The study showed that 1158 Cochrane reviews had been used in 238 current guidelines from those developers.59 Other studies about the development of guidelines have been critical of some guideline developers, suggesting that the evidence on which recommendations are made is sometimes prone to bias.60 61 62 63 Widespread use of robust and reliable evidence, such as IPD meta-analyses where available, may go a long way to averting such criticisms.
A recent systematic review of possible factors preventing uptake of evidence from systematic reviews by decision makers cited a lack of awareness and lack of familiarity as two of the major barriers.64 For IPD meta-analyses in particular, lack of familiarity or understanding of the methodology may hinder uptake. There may also be additional potential barriers preventing uptake, which may warrant further investigation. Thus for authors of IPD meta-analyses, publication alone is not enough to ensure impact on clinical practice. Many of the meta-analyses included in our sample were published in high ranking medical journals and yet did not seem to have been considered in clinical guidelines. Authors of IPD meta-analyses need to be more proactive in making guideline developers aware of new results. Current work by the UK Cochrane Centre aims to ensure that guideline developers and Cochrane review groups work in a more streamlined way to improve uptake of new Cochrane reviews. Broader dissemination, such as production of policy briefs, press releases, and increased use of social media to accompany research findings may also be beneficial. Improving the understanding of often complex methods and findings within the reports of IPD meta-analyses is needed. Clearer reporting of such meta-analyses is also necessary to ensure that the users of the reviews are better able to distinguish IPD meta-analyses from other systematic reviews and to facilitate access to methods and results arising from these reviews. This should be helped by the extension of the preferred reporting items for systematic reviews and meta-analyses (PRISMA) standards for IPD meta-analysis.
Meaning and implications
It seems clear from our survey that the use of IPD meta-analyses in clinical guidelines can, and should, be improved. Guideline developers commonly carry out their own new systematic reviews and meta-analyses to support their guideline recommendations. Indeed, this is the WHO recommendation to its guideline developers.65 Although this ensures an up to date review of reported trials, when a recent and relevant IPD meta-analysis is available, it may also mean that the most robust evidence available is overlooked. One reason for this may be poorer understanding or appreciation of IPD versus standard systematic reviews. This problem has been recognised recently by the Cochrane IPD Meta-Analysis Methods Group,10 which has developed a guidance paper to better enable users of IPD meta-analyses to understand them and to appraise their quality. Where guideline developers do use the findings of such meta-analyses in preference to standard aggregate data systematic reviews, this may lead to a better or more nuanced appreciation of the effect of the intervention and may better inform the resulting guideline. In the sample of guidelines that we assessed, however, we found few examples of this having occurred.
Where IPD meta-analyses are available, of good quality, and up to date, they should be used to inform evidence based clinical guidelines and thus clinical practice more systematically than seems to be the case from this study. Increased use of such meta-analyses could improve guidelines and ensure that routine patient care is based on the most reliable evidence available.
What is already known on this topic
Systematic reviews that utilise individual participant data (IPD) are considered to be the gold standard in evidence synthesis
Systematic reviews and meta-analyses of IPD may provide key evidence to inform clinical guideline recommendations
The extent to which IPD meta-analyses are taken up in clinical guidelines was unclear
What this study adds
Systematic reviews using IPD are being underused in current clinical guidelines
Authors of reviews using IPD could be more proactive in disseminating key findings to increase their uptake in clinical guidelines
Guidelines should report key methods and evidence used to underpin treatment recommendations consistently and transparently
Cite this as: BMJ 2015;350:h1088
We thank the attendees of the workshop on the use of IPD in systematic reviews (25 September 2012, London, UK). Attendees were drawn from members of the Cochrane IPD Meta-analysis Methods Group and those listed contributed to discussion at the workshop and provided examples of impact of IPD systematic reviews on guidelines.
P Alderson, Oxford, UK; L Askie, Sydney, Australia; D Bennett, Oxford, UK; S Burdett, London, UK; M Clarke, Belfast, UK; S Dias, Bristol, UK; J Emberson, Oxford, UK; F Gueyffier, Lyon, France; A Iorio, Ontario, Canada; M Macleod, Edinburgh, UK; BW Mol, Amsterdam, Netherlands; C Moons, Utrecht, Netherlands; M Parmar, London, UK; R Perera, Oxford, UK; R Phillips, York, UK; J P Pignon, Villejuif, France; J Rees, Bristol, UK; H Reitsma, Amsterdam, Netherlands; R Riley, Birmingham, UK; M Rovers, Nijmegan, Netherlands; L Rydzewska, London, UK; C Schmid, Boston, USA; S Shepperd, Oxford, UK; S Stenning, London, UK; L Stewart, York, UK; J Tierney, London, UK; C Tudur Smith, Liverpool, UK; C Vale, London, UK; J Welge, Cincinnati, USA; I White, Cambridge, UK; and W Whiteley, Edinburgh, UK
Contributors: LHMR, CLV, LAS, MMR, JRE, and FG conceived the study. LHMR and CLV carried out the searches, extracted and collated the data, and wrote the first draft of the manuscript. LAS, MMR, JRE, and FG contributed to the interpretation of the results and revising of the article for important intellectual content. All authors read and approved the final manuscript. LHMR and CLV are the guarantors for the study.
Funding: This work was supported by a grant from the MRC Network of Hubs for Trials Methodology Research (R20). The funder had no role in the design, analysis, interpretation of the results, or the writing of the manuscript.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation with competing interests for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work. Although there are no conflicts relating to this submission, FG has received grants from Teva and BMS, and personal fees from NovaDiscovery.
Ethical approval: Not required.
Data sharing: No additional data available.
Transparency: The lead author (CLV) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by/.