Intended for healthcare professionals

  1. Research
  2. Uptake of systematic...
  3. Uptake of systematic reviews and meta-analyses based on individual participant data in clinical practice guidelines: descriptive study
CCBYNC Open access
Research

Uptake of systematic reviews and meta-analyses based on individual participant data in clinical practice guidelines: descriptive study

BMJ 2015; 350 doi: https://doi.org/10.1136/bmj.h1088 (Published 06 March 2015) Cite this as: BMJ 2015;350:h1088
  1. Claire L Vale, senior research scientist1,
  2. Larysa H M Rydzewska, research scientist1,
  3. Maroeska M Rovers, professor of evidence based surgery2,
  4. Jonathan R Emberson, senior statistician3,
  5. François Gueyffier, professor of clinical pharmacology45,
  6. Lesley A Stewart, director6
  7. on behalf of the Cochrane IPD Meta-analysis Methods Group
  1. 1MRC Clinical Trials Unit at UCL, London WC2B 6NH, UK
  2. 2Radboud Institute for Health Sciences, Radboud University Medical Centre, Nijmegen, Netherlands
  3. 3Clinical Trial Service Unit and Epidemiological Studies Unit (CTSU), University of Oxford, Oxford, UK
  4. 4Université Claude Bernard Lyon 1, Université de Lyon, Lyon, France
  5. 5Hospices Civils de Lyon, Service de Pharmacologie Clinique & Essais Thérapeutiques, Bron cedex, France
  6. 6Centre for Reviews and Dissemination, A/B Block Alcuin College, University of York, York, UK
  1. Correspondence to: L H M Rydzewska larysa.rydzewska{at}ucl.ac.uk
  • Accepted 26 January 2015

Abstract

Objective To establish the extent to which systematic reviews and meta-analyses of individual participant data (IPD) are being used to inform the recommendations included in published clinical guidelines.

Design Descriptive study.

Setting Database maintained by the Cochrane IPD Meta-analysis Methods Group, supplemented by records of published IPD meta-analyses held in a separate database.

Population A test sample of systematic reviews of randomised controlled trials that included a meta-analysis of IPD, and a separate sample of clinical guidelines, matched to the IPD meta-analyses according to medical condition, interventions, populations, and dates of publication.

Data extraction Descriptive information on each guideline was extracted along with evidence showing use or critical appraisal, or both, of the IPD meta-analysis within the guideline; recommendations based directly on its findings and the use of other systematic reviews in the guideline.

Results Based on 33 IPD meta-analyses and 177 eligible, matched clinical guidelines there was evidence that IPD meta-analyses were being under-utilised. Only 66 guidelines (37%) cited a matched IPD meta-analysis. Around a third of these (n=22, 34%) had critically appraised the IPD meta-analysis. Recommendations based directly on the matched IPD meta-analyses were identified for only 18 of the 66 guidelines (27%). For the guidelines that did not cite a matched IPD meta-analysis (n=111, 63%), search dates had preceded the publication of the IPD meta-analysis in 23 cases (21%); however, for the remainder, there was no obvious reasons why the IPD meta-analysis had not been cited.

Conclusions Our results indicate that systematic reviews and meta-analyses based on IPD are being under-utilised. Guideline developers should routinely seek good quality and up to date IPD meta-analyses to inform guidelines. Increased use of IPD meta-analyses could lead to improved guidelines ensuring that routine patient care is based on the most reliable evidence available.

Introduction

Systematic reviews have enormous potential value in developing healthcare policy and clinical guidelines, as well as directly providing clear and reliable evidence to those making clinical decisions. Individual participant data (IPD) meta-analyses are commonly described as the gold standard of systematic reviews. Unlike systematic reviews and meta-analyses based on aggregate data (which is usually extracted from study reports) the results of IPD meta-analyses are not so restricted by many of the well documented publication or reporting biases that can hamper other non-IPD systematic reviews.1 2 3 4 5 They often include greater numbers of patients from more trials, both published and unpublished, providing more reliable summaries of trial results. Furthermore, using comprehensive IPD can generate more detailed results. For example, the data can be better used to test whether the effect of a given treatment differs in one type of patients compared with another (treatment-covariate interactions), thus potentially enabling better targeting of treatments within specific patient populations. Detailed information on the exact timing of particular outcomes also allows for a proper assessment of how any treatment benefits (or hazards) might evolve with duration of treatment (treatment-time interactions). Consequently, systematic reviews and meta-analyses of IPD can provide key evidence to inform and influence clinical guidance.

Yet despite this clear potential, the extent to which IPD meta-analyses, are actually used in practice by guideline developers, as best or level 1 evidence,6 is uncertain. Therefore, following an international workshop of 31 people drawn from the members of the Cochrane IPD Meta-analysis Methods Group (September 2012, London), we assessed whether and how such meta-analyses are used in published clinical guidelines.

We identified where practice recommendations had been based on evidence from IPD meta-analyses and ascertained whether findings from such meta-analyses were given precedence over systematic reviews using aggregated data and dealing with the same question, given the known advantages of using IPD.7 8 9 Our ultimate intention was to help inform those conducting IPD meta-analyses and those developing guidance, on how to optimise the uptake and use of these meta-analyses.

Methods

Study sample identification

Systematic reviews and meta-analyses of IPD—We extracted these from a database maintained by the Cochrane IPD Meta-analysis Methods Group,10 supplemented by records of published IPD meta-analyses held in a separate database.11 To be included in our study, the meta-analyses should have been published either in the Cochrane Library12 or in a peer reviewed journal between 1 January 2008 and 31 December 2010 and carried out a meta-analysis of IPD as part of a full systematic review of randomised controlled trials. They should have evaluated at least one treatment intervention, irrespective of healthcare area or geographical region.

Clinical guidelines—We searched the US National Guideline Clearinghouse;13 the Scottish Intercollegiate Guidelines Network,14 and the UK National Institute for Health and Care Excellence15 websites for all current clinical guidelines in healthcare areas relating to our sample of IPD meta-analyses. For guidelines related to cancer treatments, we also searched the Standards and Guidelines Evidence directory of cancer guidelines.16 Eligible guidelines should have been published either originally or in an updated form between 1 January 2010 and 31 December 2013, a timeframe that was chosen to allow for the inclusion of IPD meta-analyses from our sample; albeit that we accepted a one year overlap between publication of guidelines and meta-analyses in 2010. This was done primarily to identify references to the IPD meta-analyses published in 2008-09. Two authors (LHMR and CLV) matched all potentially eligible guidelines to the IPD meta-analyses included in the sample. Guidelines were matched to the meta-analyses if they dealt with the same medical condition, treatment type, and patient populations (for example, adults or children). We also included guidelines with a broad scope that encompassed the subject area of the meta-analysis. Matched guidelines were necessarily published after the publication date of the meta-analysis. Duplicate guidelines (that is, those identified from more than one source) were removed. Two authors (LHMR and CLV) resolved queries on the suitability of individual guidelines for inclusion in the sample by discussion.

Data extraction

For each matched clinical guideline, we extracted information on the guideline developer, date of publication, and free availability of the guideline. We also carried out a thorough text search of each guideline to identify any reference to the matched IPD meta-analyses, using search terms including the authors’ names, name of intervention, systematic review, meta-analysis, individual patient or participant data, and IPD. Furthermore, we sought data on whether any other IPD meta-analysis or meta-analysis of aggregate data was cited, and any key recommendations within the guideline that were attributed to a matched IPD meta-analysis. Two authors (LHMR and CLV) resolved any discrepancies by discussion.

Analysis

For each meta-analysis in the test sample, we recorded the total number and proportion of unique, matched guidelines that cited it. For each cited meta-analysis, we looked for evidence within the guideline that it had been critically appraised and whether specific practice recommendations had been based on the findings of the meta-analysis. We also noted whether any other related IPD meta-analyses or meta-analyses based on aggregate data had been cited, and where both types of meta-analyses were cited we looked for evidence that any distinction had been made between the two. Finally, for guidelines that did not cite the matched IPD meta-analysis, we looked for any likely reasons why it had not been included or for information about alternative evidence sources that had been used to inform the guideline recommendations.

To obtain overall totals, averages, and frequencies on which interpretation could be based we collated data from each of the IPD meta-analyses across our test sample, including the frequency of citations and the number of guidelines that based their recommendations on results of a matched IPD meta-analysis.

Results

Sample selection

Our two chosen data sources yielded 33 eligible IPD meta-analyses published between 2008 and 2010 (fig 1⇓ and table⇓). Fourteen were reviews of cancer treatments and eight were reviews of interventions related to cardiovascular disorders, mainly stroke and heart disease. The remaining 11 reviews covered a broad range of healthcare areas, including epilepsy, gynaecological disorders, otitis media, and asthma.

Figure1

Fig 1 Identification of eligible IPD meta-analyses

Description of individual patient data meta-analyses included in sample

View this table:

Quality of included meta-analyses

Each IPD meta-analysis was conducted as part of a full systematic review. In general, across the 33 meta-analyses, the systematic searches had looked across numerous sources (bibliographic databases, Cochrane Library, conference proceedings, and reference lists). Many (21/33) had also searched trials registers or consulted with relevant experts to identify unpublished or ongoing trials, of which 13 had identified and included data from one or more unpublished trials. Most of the meta-analyses (26/33) reported either having conducted a quality assessment of the included trials, either using a recognised tool (for example, Cochrane risk of bias tool)17 or thoroughly checking and verifying the validity of the data received. Where reported, the IPD meta-analyses were able to include most of the trials identified (median 86%, range 56-100%) and also of the known randomised patients (median 92%, range 65-100%). Methods of analyses used were generally well described and in keeping with the outcomes being analysed, and many had prespecified and reported patient level subgroup analyses in addition to the primary and secondary outcomes.

Guideline sample selection

Our search of clinical guidelines published between 2010 and 2013 yielded 1171 records (fig 2⇓); however, many of these (n=432, 37%) were identified as out of scope based on their titles or guideline summaries, or both (for example, unrelated disease or treatment, or both, or outside the date range) and were excluded. Duplicate records obtained from more than one guideline source, (n=201, 17%) were also discarded. The remaining 538 potentially eligible guidelines were examined in detail. Of these, a further 308 were either found to be ineligible (fig 2) or were excluded for other reasons (n=53)—for example, because a new guideline had superseded the original (n=18) or because the guideline was not freely available (n=7). Therefore, results are based on 177 guidelines published between 2010 and 2013 in clinical areas and populations matching those in the sample of IPD meta-analysis.

Figure2

Fig 2 Identification of eligible guidelines

The 177 guidelines had been written by 43 (see supplementary appendix 1) different guideline development groups (range 1-8 per developer) and spanned most geographical areas, albeit half (n=89, 50%) were North American guidelines. A further 36% (n=64) were from the United Kingdom and Europe, with the remainder being largely from Australia, New Zealand, and Asia. A matched IPD meta-analysis was cited in 66 of the 177 guidelines (37%). A third of these guidelines (n=22, 34%) had included some critical appraisal of the matched meta-analysis. The box and supplementary table 1 provide examples of guideline recommendations that could be directly attributed to the findings of cited IPD meta-analyses.

Examples of guideline recommendations linked to findings of cited IPD meta-analyses

Example 1*: concomitant chemoradiation for cervical cancer22
Potential impact—making recommendations about specific treatment regimens
The IPD meta-analysis

  • 15 randomised trials, 3452 patients

Pertinent results

  • 6% survival benefit at five years associated with chemoradiation

  • No suggestion of a difference between the treatment benefit, whether platinum based or non-platinum based treatment regimens

Recommendations in guidelines citing IPD meta-analysis

  • Concomitant chemoradiation represents the standard

  • Non-platinum based regimens for chemoradiation seem to be as efficient as platinum based chemotherapy

  • The role of adjuvant chemotherapy after concomitant chemoradiation remains unclear and should be included in further clinical investigations

Versus recommendations in guideline not citing IPD meta-analysis

  • In women with late cervical cancer: combined chemotherapy and radiotherapy is the treatment of choice.

Example 2†: antibiotics for acute otitis media54
Impact—targeting treatment to patient groups most likely to benefit
The IPD meta-analysis

  • Six randomised trials, 1643 children aged from 6 months to 12 years

Pertinent results

  • In children younger than 2 years with bilateral acute otitis media, 55% of controls and 30% receiving antibiotics still had pain, fever, or both at 3-7 days, with a rate difference between these groups of −25% (95% confidence interval −36% to −14%)

  • In children with otorrhoea the rate difference was −36% (95% confidence interval −53% to −19%)

Impact on guidelines

  • The UK National Institute for Health and Care Excellence (NICE) guideline on the treatment of otitis media recommended that, in general, antibiotics should not be prescribed, or prescription should be delayed. However, immediate antibiotic prescription was recommended for children between the ages of 6 months and 2 years with bilateral acute otitis media, and for children of any age with a diagnosis of otorrhoea and acute otitis media. Previous systematic reviews identified by the guideline developers, based on summary data, had not shown any differences between patient subgroups.

Example 3†: monotherapy for epilepsy55
Impact—providing evidence on appropriate outcome measures
The IPD meta-analysis

  • 20 randomised trials, 6418 patients with epilepsy treated with monotherapy

  • Network meta-analysis comparing eight antiepileptic drugs

  • Primary outcomes were time to treatment failure and time to 12 month remission from seizures; secondary outcome was time to first seizure

Pertinent results

  • For partial onset seizures (n=4628 (72%) patients), lamotrigine, carbamazepine, and oxcarbazepine provide the best combination of seizure control and treatment failure

  • For generalised onset tonic clonic seizures (n=1790 (28%) patients) estimates suggest valproate or phenytoin may provide the best combination of seizure control and treatment failure

Impact on guidelines

  • Without results from the IPD meta-analysis, the guideline produced by NICE would have been restricted to outcomes reported in the individual trials—for example, proportion of seizure-free participants. In contrast, the IPD meta-analysis provided robust evidence on more informative outcomes—for example, time to treatment failure, time to 12 month remission from seizures, which were used to inform the guideline recommendation

  • *Information missed when relevant IPD meta-analyses are omitted from guidelines.

  • †Information gained when relevant IPD meta-analyses are assimilated in guidelines.

Of the 177 guidelines, 69 (39%) cited a systematic review and meta-analysis of aggregate data either alongside the matched IPD meta-analysis (n=29, 42%) or alone. Where both were cited, few guidelines (n=4, 14%) made any distinction between the two systematic reviews—for example, by explicitly stating that one was based on individual patient data. There was no evidence that IPD meta-analyses were more or less likely to be included in guidelines from particular geographical locations. We saw no clear evidence that the uptake of individual IPD meta-analyses depended on the length of time since publication. However, the time to uptake in a specific guideline will vary according to the frequency with which the guidelines are produced or updated. In addition, certain guideline developers seemed more likely than others to cite IPD meta-analyses. More than half of the citations of matched IPD meta-analyses (35/66, 53%) were included in guidelines produced by only seven of the 43 guideline developers in the sample (see supplementary figure 1), whereas 13/43 guideline development groups (responsible for 31/177 guidelines) did not cite any of the eligible meta-analyses.

Matched guidelines were identified for 29 of the 33 meta-analyses included in the sample, with a median of five guidelines for each meta-analysis. The highest number of matched guidelines was for one meta-analysis in non-small cell lung cancer, for which 16 matched guidelines were identified. Of the 29 meta-analyses with matched guidelines, 21 (72%) had been cited in at least one guideline (median 3, range 1-14 guidelines) and eight (28%) were not cited in any of the matched guidelines. Of the 21 meta-analyses that were cited, only five were mentioned in more than half of the relevant guidelines (table 1).

Importantly, around two thirds of the guidelines in our sample (n=111, 63%) did not cite any IPD meta-analysis, despite one being available in every case. Three of these guidelines (2%) had referenced another guideline as their primary evidence source, and on further investigation it was clear that the guideline referenced had used an IPD meta-analysis. For 23 guidelines (21%) we found that the search dates had preceded the publication date of the IPD meta-analysis. A further 11 guidelines (10%) did not supply their evidence sources in a reference list and so we were unable to tell what had led to their recommendations. However, for the rest (n=74, 67%) we could find no obvious reasons why the IPD meta-analysis had not been cited. Although we did not carry out any formal quality appraisal for the included guidelines, we did seek information from either the individual guideline developers’ handbooks or their websites as to whether they advocated the use of the appraisal of guidelines for research and evaluation (AGREE) tool51 52 for guideline development or the grading of recommendations assessment, development, and evaluation (GRADE)53 system for appraising the quality of the included evidence. Of the 43 guideline development groups, we found that 14 guideline developers explicitly promoted the use of either the AGREE or the GRADE tools in their guidance documents and six referred to other methods of appraisal. However, for the remainder there was either no mention of the standards adhered to in the information available to us (n=5) or no information could be found (n=18).

Discussion

Overall, based on this sample of 33 IPD meta-analyses and 177 matched guidelines, our results indicate that systematic reviews, based on IPD, are being under-utilised as a source of the best available evidence within clinical guidelines. Little more than a third of the guidelines in our sample had cited a recent and relevant IPD meta-analysis, and less than 20% of the guidelines had clearly used information from the meta-analysis in formulating their recommendations. Where such meta-analyses had been cited as part of the evidence base, there was often little or no distinction between an IPD meta-analysis and other non-IPD (or non-systematic) reviews or individual randomised controlled trials, which were often all assigned the highest level of evidence (for example, 1++ or 1A). It was also apparent that certain guideline developers were more likely to use IPD meta-analyses as the basis of their recommendations, as more than half of all the guidelines in our sample that cited a meta-analysis were produced by only seven guideline development groups. It is also unclear why a high proportion of guidelines utilised evidence from existing systematic reviews of aggregate data (or carried out their own analyses based on aggregate data) when relevant systematic reviews and meta-analyses of IPD were available, as was the case for each of the guidelines included in this study. We believe that a relevant IPD meta-analysis should always contribute to clinical guidelines if available, unless it is determined to be of poor methodological quality, based on limited data or superseded by a more recent aggregate data meta-analysis that includes several new trials. None of these exceptions occurred in our sample. Furthermore, even when a more recent aggregate data review is available, pre-existing IPD meta-analyses may still provide important additional information—for example, relating to the effects of an intervention in different subgroups of patients, thus making an important contribution to the overall evidence base. Failure to assimilate information from IPD meta-analyses may lead to less appropriate or limited recommendations being made, whereas their uptake may better inform the guideline recommendations (see box).

Strengths and weaknesses of this study

Ours was not designed to be a comprehensive study but rather to provide insight into the uptake of IPD meta-analyses by guideline developers. Therefore we are aware of certain limitations. Firstly, we restricted our searches to three main sources (four for cancer guidelines) and so may not have identified all relevant guidelines. That many of the IPD meta-analyses identified were of treatment interventions for cancer (14/33), reflects the tradition of IPD meta-analysis in cancer research. We also searched an additional source of cancer specific guidelines. While we accept that the sample may not be entirely representative, we screened more than 1100 guidelines in total and included guidelines produced by several high profile organisations. Secondly, the interval between publication of an IPD meta-analysis and its subsequent uptake in a guideline can vary considerably across guideline development groups. Intervals depend on updating schedules or triggers and on available resources, and usually range from six months to two years. Therefore, although we considered the use of a maximum three year time lag between publication of the meta-analysis (2008-10) and publication of the matched guideline (2010-13) as an appropriate interval, this does not guarantee that we identified all relevant citations. However, we found that for the meta-analyses within this sample, many were incorporated in guidelines within a year of publication. Thirdly, a lack of consistency in the reporting or presentation of guidelines made it difficult to consistently attribute recommendations to the evidence supporting them. For example, some guidelines omitted details of their methods for searching the literature, or provided sparse (or no) reference lists. Therefore it was not always clear if a relevant meta-analysis had not been identified in the searches carried out or whether it would have been identified but not included. This may reflect fundamental differences in the quality of the included guidelines. Although we did not carry out a formal quality assessment of all of the included guidelines, we found that of the seven guideline development groups with guidelines that included more than half of the citations of IPD meta-analyses that we identified, five indicated that at an organisational level they advocated use of either the AGREE tool51 52 (Alberta Health Services, American College of Cardiology Foundation/American Heart Association, American College of Chest Physicians, and Scottish Intercollegiate Guideline Network, SIGN) or the Agency for Healthcare Research and Quality guidelines56 57 (American College of Radiology). Full reporting of evidence sources and methodology used may better enable users to appraise the quality of the guideline.58 Finally, a small number of potentially eligible guidelines were not freely available. While we were able to obtain some of these through our institution’s library, others were not accessible and could not be included. None the less, we believe that the data underpinning our review are robust enough to enable us to draw qualitatively reliable conclusions about the use of IPD meta-analyses in guidelines.

Relation to other studies

To our knowledge this is the first review focussing specifically on the inclusion of IPD meta-analyses in clinical guidelines. With recent calls to improve and increase access to trial data, the number of available such meta-analyses may increase dramatically. Therefore we considered that this study provides an important insight into the current situation on the assimilation of IPD meta-analyses into clinical guidelines. However, others have looked at uptake of systematic reviews in guidelines more generally. For example, recent results from ongoing work at the UK Cochrane Centre have shown that Cochrane reviews across the breadth of healthcare areas covered by the collaboration have been used to inform findings on the background of clinical guidelines produced by NICE, SIGN, and the World Health Organization. The study showed that 1158 Cochrane reviews had been used in 238 current guidelines from those developers.59 Other studies about the development of guidelines have been critical of some guideline developers, suggesting that the evidence on which recommendations are made is sometimes prone to bias.60 61 62 63 Widespread use of robust and reliable evidence, such as IPD meta-analyses where available, may go a long way to averting such criticisms.

A recent systematic review of possible factors preventing uptake of evidence from systematic reviews by decision makers cited a lack of awareness and lack of familiarity as two of the major barriers.64 For IPD meta-analyses in particular, lack of familiarity or understanding of the methodology may hinder uptake. There may also be additional potential barriers preventing uptake, which may warrant further investigation. Thus for authors of IPD meta-analyses, publication alone is not enough to ensure impact on clinical practice. Many of the meta-analyses included in our sample were published in high ranking medical journals and yet did not seem to have been considered in clinical guidelines. Authors of IPD meta-analyses need to be more proactive in making guideline developers aware of new results. Current work by the UK Cochrane Centre aims to ensure that guideline developers and Cochrane review groups work in a more streamlined way to improve uptake of new Cochrane reviews. Broader dissemination, such as production of policy briefs, press releases, and increased use of social media to accompany research findings may also be beneficial. Improving the understanding of often complex methods and findings within the reports of IPD meta-analyses is needed. Clearer reporting of such meta-analyses is also necessary to ensure that the users of the reviews are better able to distinguish IPD meta-analyses from other systematic reviews and to facilitate access to methods and results arising from these reviews. This should be helped by the extension of the preferred reporting items for systematic reviews and meta-analyses (PRISMA) standards for IPD meta-analysis.

Meaning and implications

It seems clear from our survey that the use of IPD meta-analyses in clinical guidelines can, and should, be improved. Guideline developers commonly carry out their own new systematic reviews and meta-analyses to support their guideline recommendations. Indeed, this is the WHO recommendation to its guideline developers.65 Although this ensures an up to date review of reported trials, when a recent and relevant IPD meta-analysis is available, it may also mean that the most robust evidence available is overlooked. One reason for this may be poorer understanding or appreciation of IPD versus standard systematic reviews. This problem has been recognised recently by the Cochrane IPD Meta-Analysis Methods Group,10 which has developed a guidance paper to better enable users of IPD meta-analyses to understand them and to appraise their quality. Where guideline developers do use the findings of such meta-analyses in preference to standard aggregate data systematic reviews, this may lead to a better or more nuanced appreciation of the effect of the intervention and may better inform the resulting guideline. In the sample of guidelines that we assessed, however, we found few examples of this having occurred.

Conclusions

Where IPD meta-analyses are available, of good quality, and up to date, they should be used to inform evidence based clinical guidelines and thus clinical practice more systematically than seems to be the case from this study. Increased use of such meta-analyses could improve guidelines and ensure that routine patient care is based on the most reliable evidence available.

What is already known on this topic

  • Systematic reviews that utilise individual participant data (IPD) are considered to be the gold standard in evidence synthesis

  • Systematic reviews and meta-analyses of IPD may provide key evidence to inform clinical guideline recommendations

  • The extent to which IPD meta-analyses are taken up in clinical guidelines was unclear

What this study adds

  • Systematic reviews using IPD are being underused in current clinical guidelines

  • Authors of reviews using IPD could be more proactive in disseminating key findings to increase their uptake in clinical guidelines

  • Guidelines should report key methods and evidence used to underpin treatment recommendations consistently and transparently

Notes

Cite this as: BMJ 2015;350:h1088

Footnotes

  • We thank the attendees of the workshop on the use of IPD in systematic reviews (25 September 2012, London, UK). Attendees were drawn from members of the Cochrane IPD Meta-analysis Methods Group and those listed contributed to discussion at the workshop and provided examples of impact of IPD systematic reviews on guidelines.

  • P Alderson, Oxford, UK; L Askie, Sydney, Australia; D Bennett, Oxford, UK; S Burdett, London, UK; M Clarke, Belfast, UK; S Dias, Bristol, UK; J Emberson, Oxford, UK; F Gueyffier, Lyon, France; A Iorio, Ontario, Canada; M Macleod, Edinburgh, UK; BW Mol, Amsterdam, Netherlands; C Moons, Utrecht, Netherlands; M Parmar, London, UK; R Perera, Oxford, UK; R Phillips, York, UK; J P Pignon, Villejuif, France; J Rees, Bristol, UK; H Reitsma, Amsterdam, Netherlands; R Riley, Birmingham, UK; M Rovers, Nijmegan, Netherlands; L Rydzewska, London, UK; C Schmid, Boston, USA; S Shepperd, Oxford, UK; S Stenning, London, UK; L Stewart, York, UK; J Tierney, London, UK; C Tudur Smith, Liverpool, UK; C Vale, London, UK; J Welge, Cincinnati, USA; I White, Cambridge, UK; and W Whiteley, Edinburgh, UK

  • Contributors: LHMR, CLV, LAS, MMR, JRE, and FG conceived the study. LHMR and CLV carried out the searches, extracted and collated the data, and wrote the first draft of the manuscript. LAS, MMR, JRE, and FG contributed to the interpretation of the results and revising of the article for important intellectual content. All authors read and approved the final manuscript. LHMR and CLV are the guarantors for the study.

  • Funding: This work was supported by a grant from the MRC Network of Hubs for Trials Methodology Research (R20). The funder had no role in the design, analysis, interpretation of the results, or the writing of the manuscript.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation with competing interests for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work. Although there are no conflicts relating to this submission, FG has received grants from Teva and BMS, and personal fees from NovaDiscovery.

  • Ethical approval: Not required.

  • Data sharing: No additional data available.

  • Transparency: The lead author (CLV) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by/.

References

  1. ↵
    Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ2010;340:c365.
    OpenUrlAbstract/FREE Full Text
  2. ↵
    Stewart L, Tierney J, Burdett S. Do systematic reviews based on individual patient data offer a means of circumventing biases associated with trial publications? In: Rothstein H, Sutton A, Borenstein M, eds. Publication bias in meta-analysis: prevention, assessment and adjustments. Chichester: Wiley, 2005:261-86.
  3. ↵
    Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic review of the empirical evidence of study publication bias and outcome reporting bias—an updated review. PLoS One2013;8:e66844.
    OpenUrlCrossRefPubMed
  4. ↵
    Tierney JF, Stewart LA. Investigating patient exclusion bias in meta-analysis. Int J Epidemiol2005;34:79-87.
    OpenUrlAbstract/FREE Full Text
  5. ↵
    Burdett S, Stewart LA, Tierney JF. Publication bias and meta-analysis (a practical example). Int J Technol Assess Health Care2003;19:129-34.
    OpenUrlCrossRefPubMedWeb of Science
  6. ↵
    Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg2011;128:305-10.
    OpenUrlCrossRefPubMed
  7. ↵
    Stewart LA, Clarke MJ, on behalf of the Cochrane Working Party Group on Meta-analysis using Individual Patient Data. Practical methodology of meta-analyses (overviews) using updated individual patient data. Stat Med1995;14:2057-79.
    OpenUrlCrossRefPubMedWeb of Science
  8. ↵
    Stewart LA, Tierney JF. To IPD or not to IPD? Advantages and disadvantages of systematic reviews using individual patient data. Eval Health Prof2002;25:76-97.
    OpenUrlAbstract/FREE Full Text
  9. ↵
    Higgins JPT, Green SJ, eds. Cochrane handbook for systematic reviews of interventions. Wiley, 2008.
  10. ↵
    Cochrane Individual Participant Data Meta-analysis Methods Group. 2014. http://ipdmamg.cochrane.org/.
  11. ↵
    Ahmed I, Sutton AJ, Riley RD. Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey. BMJ2012;344:d7762.
    OpenUrlCrossRefPubMed
  12. ↵
    Cochrane Database of Systematic Reviews (in the Cochrane Library) 2014. www.thecochranelibrary.com.
  13. ↵
    Agency for Healthcare Research and Quality—National Guideline Clearinghouse (NGC). 2014. www.guideline.gov/.
  14. ↵
    Scottish Intercollegiate Guideline Network (SIGN). 2014. www.sign.ac.uk/.
  15. ↵
    National Institute for Health and Care Excellence (NICE). 2014. http://guidance.nice.org.uk/.
  16. ↵
    Standards and Guidelines Evidence (SAGE). 2014. www.cancerview.ca/cv/portal/Home/ResourceLibrary.
  17. ↵
    Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ2011;343:d5928.
    OpenUrlFREE Full Text
  18. Early Breast Cancer Trialists’ Collaborative Group, Correa C, McGale P, Taylor C, Wang Y, Clarke M, et al. Overview of the randomized trials of radiotherapy in ductal carcinoma in situ of the breast. J Natl Cancer Inst Monogr2010;2010:162-77.
    OpenUrlAbstract/FREE Full Text
  19. Early Breast Cancer Trialists’ Collaborative Group, Clarke M, Coates AS, Darby SC, Davies C, Gelber RD, et al. Adjuvant chemotherapy in oestrogen-receptor-poor breast cancer: patient-level meta-analysis of randomised trials. Lancet2008;371:29-40.
    OpenUrlCrossRefPubMedWeb of Science
  20. Dowsett M, Cuzick J, Ingle J, Coates A, Forbes J, Bliss J, et al. Meta-analysis of breast cancer outcomes in adjuvant trials of aromatase inhibitors versus tamoxifen. J Clin Oncol2010;28:509-18.
    OpenUrlAbstract/FREE Full Text
  21. Piccart-Gebhart MJ, Burzykowski T, Buyse M, Sledge G, Carmichael J, Luck HJ, et al. Taxanes alone or in combination with anthracyclines as first-line therapy of patients with metastatic breast cancer. J Clin Oncol2008;26:1980-6.
    OpenUrlAbstract/FREE Full Text
  22. ↵
    Chemoradiotherapy for Cervical Cancer Meta-analysis Collaboration. Reducing uncertainties about the effects of chemoradiotherapy for cervical cancer: a systematic review and meta-analysis of individual patient data from 18 randomized trials. J Clin Oncol2008;26:5802-12.
    OpenUrlAbstract/FREE Full Text
  23. Eden T, Pieters R, Richards S, Childhood Acute Lymphoblastic Leukaemia Collaborative Group. Systematic review of the addition of vincristine plus steroid pulses in maintenance treatment for childhood acute lymphoblastic leukaemia - an individual patient data meta-analysis involving 5,659 children. Br J Haematol2010;149:722-33.
    OpenUrlCrossRefPubMed
  24. Childhood Acute Lymphoblastic Leukaemia Collaborative Group. Beneficial and harmful effects of anthracyclines in the treatment of childhood acute lymphoblastic leukaemia: a systematic review and meta-analysis. Br J Haematol2009;145:376-88.
    OpenUrlCrossRefPubMed
  25. Uronis HE, Currow DC, McCrory DC, Samsa GP, Abernethy AP. Oxygen for relief of dyspnoea in mildly- or non-hypoxaemic patients with cancer: a systematic review and meta-analysis. Br J Cancer2008;98:294-9.
    OpenUrlCrossRefPubMedWeb of Science
  26. Baujat B, Bourhis J, Blanchard P, Overgaard J, Ang KK, Saunders M, et al. Hyperfractionated or accelerated radiotherapy for head and neck cancer. Cochrane Database Syst Rev2010;12:CD002026.
    OpenUrlPubMed
  27. Pignon JP, le Maitre A, Maillard E, Bourhis J. Meta-analysis of chemotherapy in head and neck cancer (MACH-NC): an update on 93 randomised trials and 17,346 patients. Radiother Oncol2009;92:4-14.
    OpenUrlCrossRefPubMedWeb of Science
  28. Greb A, Bohlius J, Schiefer D, Schwarzer G, Schulz H, Engert A. High-dose chemotherapy with autologous stem cell transplantation in the first line treatment of aggressive non-Hodgkin lymphoma (NHL) in adults. Cochrane Database Syst Rev2008;1:CD004024.
    OpenUrlPubMed
  29. Auperin A, Le Pechoux C, Rolland E, Curran WJ, Furuse K, Fournel P, et al. Meta-analysis of concomitant versus sequential radiochemotherapy in locally advanced non-small-cell lung cancer. J Clin Oncol2010;28:2181-90.
    OpenUrlAbstract/FREE Full Text
  30. NSCLC Meta-analyses Collaborative Group. Adjuvant chemotherapy, with or without postoperative radiotherapy, in operable non-small cell lung cancer: two meta-analyses of individual patient data. Lancet2010;375:1267-77.
    OpenUrlCrossRefPubMedWeb of Science
  31. NSCLC Meta-analyses Collaborative Group. Chemotherapy in addition to supportive care improves survival in advanced non-small-cell lung cancer: a systematic review and meta-analysis of individual patient data from 16 randomized controlled trials. J Clin Oncol2008;26:4617-25.
    OpenUrlAbstract/FREE Full Text
  32. De Luca G, Gibson CM, Bellandi F, Murphy S, Maioli M, Noc M, et al. Early glycoprotein IIb-IIIa inhibitors in primary angioplasty (EGYPT) cooperation: an individual patient data meta-analysis. Heart2008;94:1548-58.
    OpenUrlAbstract/FREE Full Text
  33. Bejan-Angoulvant T, Saadatian-Elahi M, Wright JM, Schron EB, Lindholm LH, Fagard R, et al. Treatment of hypertension in patients 80 years and older: the lower the better? A meta-analysis of randomized controlled trials. J Hypertens2010;28:1366-72.
    OpenUrlCrossRefPubMedWeb of Science
  34. De Backer TL, Vander Stichele R, Lehert P, Van Bortel L. Naftidrofuryl for intermittent claudication. Cochrane Database Syst Rev2008;2:CD001368.
    OpenUrlPubMed
  35. Cholesterol Treatment Trialists (CTT) Collaboration, Baigent C, Blackwell L, Emberson J, Holland LE, Reith C, et al. Efficacy and safety of more intensive lowering of LDL cholesterol: a meta-analysis of data from 170,000 participants in 26 randomised trials. Lancet2010;376:1670-81.
    OpenUrlCrossRefPubMedWeb of Science
  36. Antithrombotic Trialists (ATT) Collaboration, Baigent C, Blackwell L, Collins R, Emberson J, Godwin J, et al. Aspirin in the primary and secondary prevention of vascular disease: collaborative meta-analysis of individual participant data from randomised trials. Lancet2009;373:1849-60.
    OpenUrlCrossRefPubMedWeb of Science
  37. Halkes PH, Gray LJ, Bath PM, Diener HC, Guiraud-Chaumeil B, Yatsu FM, et al. Dipyridamole plus aspirin versus aspirin alone in secondary prevention after TIA or stroke: a meta-analysis by risk. J Neurol Neurosurg Psychiatry2008;79:1218-23.
    OpenUrlAbstract/FREE Full Text
  38. Craig LE, Bernhardt J, Langhorne P, Wu O. Early mobilization after stroke: an example of an individual patient data meta-analysis of a complex intervention. Stroke2010;41:2632-6.
    OpenUrlAbstract/FREE Full Text
  39. Ellis G, Mant J, Langhorne P, Dennis M, Winner S. Stroke liaison workers for stroke patients and carers: an individual patient data meta-analysis. Cochrane Database Syst Rev2010;5:CD005066.
    OpenUrlPubMed
  40. Browning GG, Rovers MM, Williamson I, Lous J, Burton MJ. Grommets (ventilation tubes) for hearing loss associated with otitis media with effusion in children. Cochrane Database Syst Rev2010;10:CD001801.
    OpenUrlPubMed
  41. Koopman L, Hoes AW, Glasziou PP, Appelman CL, Burke P, McCormick DP, et al. Antibiotic therapy to prevent the development of asymptomatic middle ear effusion in children with acute otitis media: a meta-analysis of individual patient data. Arch Otolaryngol Head Neck Surg2008;134:128-32.
    OpenUrlCrossRefPubMed
  42. Kassai B, Chiron C, Augier S, Cucherat M, Rey E, Gueyffier F, et al. Severe myoclonic epilepsy in infancy: a systematic review and a meta-analysis of individual patient data. Epilepsia2008;49:343-8.
    OpenUrlCrossRefPubMed
  43. Ford AC, Moayyedi P, Jarbol DE, Logan RF, Delaney BC. Meta-analysis: Helicobacter pylori ‘test and treat’ compared with empirical acid suppression for managing dyspepsia. Aliment Pharmacol Ther2008;28:534-44.
    OpenUrlCrossRefPubMed
  44. Daniels JP, Middleton L, Xiong T, Champaneria R, Johnson NP, Lichten EM, et al. Individual patient data meta-analysis of randomized evidence to assess the effectiveness of laparoscopic uterosacral nerve ablation in chronic pelvic pain. Hum Reprod Update2010;16:568-76.
    OpenUrlAbstract/FREE Full Text
  45. Middleton LJ, Champaneria R, Daniels JP, Bhattacharya S, Cooper KG, Hilken NH, et al. Hysterectomy, endometrial destruction, and levonorgestrel releasing intrauterine system (Mirena) for heavy menstrual bleeding: systematic review and meta-analysis of data from individual patients. BMJ2010;341:c3929.
    OpenUrlAbstract/FREE Full Text
  46. Cools F, Askie LM, Offringa M, Asselin JM, Calvert SA, Courtney SE, et al. Elective high-frequency oscillatory versus conventional ventilation in preterm infants: a systematic review and meta-analysis of individual patients’ data. Lancet2010;375:2082-91.
    OpenUrlCrossRefPubMedWeb of Science
  47. Cates CJ, Cates MJ. Regular treatment with salmeterol for chronic asthma: serious adverse events. Cochrane Database Syst Rev2008;3:CD006363.
    OpenUrlPubMed
  48. Young J, De Sutter A, Merenstein D, van Essen GA, Kaiser L, Varonen H, et al. Antibiotics for adults with clinically diagnosed acute rhinosinusitis: a meta-analysis of individual patient data. Lancet2008;371:908-14.
    OpenUrlCrossRefPubMedWeb of Science
  49. Wang D, Connock M, Barton P, Fry-Smith A, Aveyard P, Moore D. ‘Cut down to quit’ with nicotine replacement therapies in smoking cessation: a systematic review of effectiveness and economic analysis. Health Technol Assess2008;12:iii-iv, ix-xi, 1-135.
    OpenUrlPubMed
  50. O’Meara S, Tierney J, Cullum N, Bland JM, Franks PJ, Mole T, et al. Four layer bandage compared with short stretch bandage for venous leg ulcers: systematic review and meta-analysis of randomised controlled trials with data from individual patients. BMJ2009;338::b1344.
  51. ↵
    AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care2003;12:18-23.
    OpenUrlAbstract/FREE Full Text
  52. ↵
    AGREE Next Steps Consortium (2009). The AGREE II Instrument. 2014. www.agreetrust.org.
  53. ↵
    Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1 introduction—grade evidence profiles and summary of findings tables. J Clin Epidemiol2011;64:383-94.
    OpenUrlCrossRefPubMed
  54. ↵
    Rovers MM, Glasziou P, Appelman CL, Burke P, McCormick DP, Damoiseaux RA, et al. Antibiotics for acute otitis media: a meta-analysis with individual patient data. Lancet2006;368:1429-35.
    OpenUrlCrossRefPubMedWeb of Science
  55. ↵
    Tudur Smith C, Marson AG, Chadwick DW, Williamson PR. Multiple treatment comparisons in epilepsy monotherapy trials. Trials2007;8:34.
    OpenUrlCrossRefPubMed
  56. ↵
    Agency for Healthcare Research and Quality—National Guideline Clearinghouse (NGC). Inclusion criteria. 2014. www.guideline.gov/about/inclusion-criteria.aspx.
  57. ↵
    Institute of Medicine. Graham R MM, Wolman DM, Greenfield S, Steinberg E, eds. Clinical practice guidelines we can trust. National Academies Press, 2011:2p. www.iom.edu/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust/Standards.aspx.
  58. ↵
    Qaseem A, Forland F, Macbeth F, Ollenschlager G, Phillips S, van der Wees P. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med2012;156:525-31.
    OpenUrlCrossRefPubMedWeb of Science
  59. ↵
    UK Cochrane Centre. Guidelines project. 2014. http://ukcccochraneorg/guidelines-project.
  60. ↵
    Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC Jr. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA2009;301:831-41.
    OpenUrlCrossRefPubMedWeb of Science
  61. ↵
    Oxman AD, Lavis JN, Fretheim A. Use of evidence in WHO recommendations. Lancet2007;369:1883-9.
    OpenUrlCrossRefPubMedWeb of Science
  62. ↵
    Vigna-Taglianti F, Vineis P, Liberati A, Faggiano F. Quality of systematic reviews used in guidelines for oncology practice. Ann Oncol2006;17:691-701.
    OpenUrlAbstract/FREE Full Text
  63. ↵
    Kmietowicz Z. A third of recommendations in Scottish guidelines are based on poorest evidence, finds study. BMJ2014;348:g1428.
    OpenUrlFREE Full Text
  64. ↵
    Wallace J, Byrne C, Clarke M. Making evidence more wanted: a systematic review of facilitators to enhance the uptake of evidence from systematic reviews and meta-analyses. Int J Evid Based Healthc2012;10:338-46.
    OpenUrlCrossRefPubMed
  65. ↵
    Global Programme on Evidence for Health Policy—World Health Organization. Guidelines for WHO guidelines. 2003. http://whqlibdoc.who.int/hq/2003/eip_gpe_eqc_2003_1.pdf.
View Abstract
Back to top