Meta-analysis: Unresolved issues and future developmentsBMJ 1998; 316 doi: https://doi.org/10.1136/bmj.316.7126.221 (Published 17 January 1998) Cite this as: BMJ 1998;316:221
- George Davey Smith, professor of clinical epidemiology ()a,
- Matthias Egger, reader in social medicine and epidemiologya
- Correspondence to: Professor Davey Smith
Since its recent introduction into clinical epidemiology, meta-analysis has established itself as an influential branch of biostatistics. Several books have focused mainly or entirely on meta-analysis in medicine,1 2 3 4 5 and the latest editions of relevant textbooks generally include a section on meta-analysis.6 7 8 9 Computer software entirely devoted to meta-analysis has been developed, and meta-analytic procedures have been introduced in general statistical software packages. We will soon be providing an overview of software packages on the BMJ's website.10 Several unresolved issues concerning meta-analysis remain, and in this final article of our series we address some of the topics that are likely to feature in future discussions of the appropriate practice and domain of meta-analysis.
Should unpublished data be included in meta-analyses?
Publication bias, discussed in detail in a previous article,11 is a major threat to the validity of meta-analysis. Obtaining and including data from unpublished studies seems to be the obvious way of avoiding this problem. Including data from unpublished studies can itself introduce bias, however. Even after extensive consultation with the research community, unpublished studies may remain hidden. The unpublished studies that can be located may thus be an unrepresentative sample of unpublished studies. Whether bias is reduced or increased by including unpublished studies cannot formally be assessed as it is impossible to be certain that all unpublished studies have been located. A further problem relates to the willingness of investigators of located unpublished studies to provide data. This may depend on the findings of the study, more favourable results being provided more readily. This could again bias the findings of a meta-analysis.
Meta-analysis has established itself as an important technique in clinical epidemiology, but several issues remain unresolved
The inclusion of unpublished, non-peer reviewed data can be problematic, particularly if these data come from interested sources, such as the pharmaceutical industry
Individual patient data are often required to address important questions, but the mechanisms to facilitate increasing availability of trial data for meta-analysis are lacking
The clinical application of results from meta-analyses to the individual patient often remains a difficult matter of judgment
The Cochrane Collaboration will have an important role in future developments in the field of systematic reviews and meta-analyses
An analysis of 150 meta-analyses published between 1988 and 1991 showed that most meta-analysts had searched for unpublished material, although such data were located and included in only 31% of meta-analyses.12 A questionnaire assessing the attitudes towards inclusion of unpublished data was sent to the authors of these reports and to the editors of the journals that had published them: 78% of meta-analysts supported the use of unpublished material, compared with only 47% of journal editors.12 This lack of support by some editors is on the grounds that the data have not been peer reviewed. The refereeing process, however, has not always been a successful way of ensuring that published results are valid.13 14 On the other hand, meta-analyses of unpublished data from interested sources is clearly of concern. Such unchallengeable data have been produced in circumstances in which an obvious commercial interest exists (box gives an example).
Box 1: Controversy over selective serotonin uptake inhibitors and depression
Selective serotonin uptake inhibitors are widely used for the treatment of depression, although their clinical advantages over the much less expensive tricyclic antidepressants have not been well established.
In their meta-analysis Song et al used the dropout rate among randomised controlled trial participants taking selective serotonin uptake inhibitors and those taking conventional antidepressants as an indicator of therapeutic success15: patients who stop taking their treatment because of inefficacy or side effects are the ones who are not benefiting, and thus the class of drug with the lower dropout rate can be considered the one with the more favourable effects.
There was little difference between selective serotonin uptake inhibitors and the other-usually tricyclic-antidepressants. In response to this analysis, Nakielny (for Lilly Industries, the manufacturers of fluoxetine) presented a meta-analysis of 14 investigational studies of new drugs which they stated included every study completed by December 1990.16 This included what were called “unpublished data on file.” The pooled dropout rates calculated by Nakielny differed markedly from the literature based analysis.
Lilly Industries claimed that its analysis was not “subject to biases introduced by selective publication and literature searches,” but this is difficult to assess if the trials included represent unpublished data “on file”. To make such data available in the future is one of the major challenges facing meta-analysts and the promoters of systematic reviews and evidence based medicine.
The most satisfactory approach to the inclusion of unpublished data in meta-analyses is to carry out an extensive search for such data and obtain them if possible. The analysis should then be performed with and without the unpublished data, as a form of sensitivity analysis. If the conclusions are altered through the inclusion or exclusion of such data, the results of either approach should be treated cautiously.
Subjectivity in data analysis and reporting
Using published results exclusively can introduce biases other than those of publication bias. The choice of the outcome that is reported can be influenced by the results: the outcome with the most favourable findings will generally be reported. An example of how published results can be misleading comes from two separate analyses of a double blind, placebo controlled trial assessing the efficacy of amoxycillin in children with non-suppurative otitis media.17 18 Opposite conclusions were reached, mainly because different weight was given to the various outcome measures assessed in the study. This disagreement was conducted in the public arena, as it was accompanied by accusations of impropriety against the team producing the findings favourable to amoxycillin. The leader of this team had received considerable funding, both in research grants and as personal honorarium, from the manufacturers of amoxycillin.19 This is a good example of how reliance on the data chosen to be presented by the investigators can lead to distortion.20 This has probably been a frequent source of bias, which only in rare occasions becomes common knowledge. With improving standards of clinical trial reporting21 subjectivity in data analysis should become less common in the future.
Individual patient data or summary statistics-which should be included in a meta-analysis?
Meta-analyses that have been entirely dependent on summary data obtained from published reports of clinical trials have provided robust indicators of treatment outcomes. Such analyses have been described as meta-analyses of the literature.22 If a researcher is interested in outcomes in different groups, however, the analysis will be made difficult if the various trials do not report data accordingly. For example, a literature based meta-analysis of the effect of drug treatment of hypertension in elderly people23 was obliged to use a definition of “elderly” that included the participants aged 60 or over from some studies and those aged 65 or over from others. Also, because many trials failed to report age stratified data, less than half of the potential trials could be included in the analysis. This could create serious bias, as the decision of investigators to publish age stratified data may have been dependent on results.
Supplementary data from individual trials are increasingly being obtained for meta-analyses. For example, by obtaining data on mortality from coronary heart disease according to grouped follow up periods from the original investigators of cholesterol lowering trials, Law et al were able to show that the reduction in risk of coronary heart disease consequent on cholesterol lowering increased with the duration of treatment.23 Several collaborative groups have assembled data on each participant within the separate trials. This greatly increases flexibility when defining groups within the different trials for subgroup analyses, and also allows use of data on the exact time to the event for each participant. For example, the Fibrinolytic Therapy Trialists’ Collaborative Group investigated the effect of thrombolysis after myocardial infarction according to (a) the electrocardiographic abnormalities of patients at entry to the study; (b) the time at which treatment was received after onset of symptoms; (c) the age and sex of the patients; and (d) the presence or absence of various comorbid conditions.24 This permits comparisons that retain the advantage of the original randomisation to be made, with the proviso that the separate trials did not necessarily use stratified randomisation according to these characteristics. box presents a further example.
Box 2: Coronary artery bypass graft surgery and survival: meta-analysis using individual patient data
It has long been accepted that coronary artery bypass graft surgery provides effective relief from angina pectoris and that it prolongs survival in high risk patients with left main artery disease
The effect of such surgery on survival in other categories of patients with coronary heart disease, however, remains controversial
A meta-analysis of trials compared coronary artery bypass graft surgery with conventional treatment in patients with stable coronary heart disease.25 The graft surgery overall was associated with a significant reduction of mortality-for example, at five years 10.2% v 15.8%, P=0.0001)
For this meta-analysis the individual patient data made it possible to perform several subgroup analyses. For example, by using a modification of the veterans administration risk score26 (which is based on the presence of class III or IV angina, ST depression at rest, history of hypertension, and history of myocardial infarction) the relation of benefit with the level of risk could be explored. No benefit was evident in the third at lowest risk, which was characterised by a relatively low five year mortality of 5.5%. Conversely, benefit was present for groups of patients at higher risk of death (1). This information is crucial to the clinical application of the results from meta-analyses, indicating that targeting coronary artery bypass graft surgery at high risk individuals would be an efficient way of using limited resources
This example illustrates how important information can be derived from risk stratification based on individual patient data
Obtaining individual patient data has advantages beyond the ability to perform standardised subgroup analyses.27 Contact with individual investigators can help to identify further trials-published and unpublished-which the meta-analysts had missed. It may be possible to identify deviations from protocols in the trials-for example, participants who were included even though they did not satisfy entry criteria. Incorrect analyses-for example, deviation from “intention to treat” analysis, the presence of unreported dropouts, and simple oversights-may be identified. Outcome measures can be better standardised across the trials, which will counteract the tendency of researchers to publish the results only in terms of the most striking effect on a particular outcome. Additional follow up data can also be obtained, as for some trials the period of randomised comparison continues beyond the initial publication, but only the published data are publicly available.
Value of “failed” meta-analyses
In some cases a conclusive meta-analysis may not be possible if methodological standards are to be maintained. In such “failed” meta analyses28 the treatment methods, concurrent treatment, length of follow up, characteristics of the study participants, or end points that were measured might be too varied to allow for the sensible combination of results. A meta-analysis exclusively based on a small number of trials will often have to be inconclusive, even if the combined estimate of effect is significant.11 29
The Cochrane Database of Systematic Reviews contains many examples of treatment interventions for which, the reviewers thought, meta-analysis had failed to produce a conclusive answer. For example, the review on thrombolysis in acute ischaemic stroke, published in the second issue of the Cochrane Library in 1996,30 stated: “…the data so far are scant, and quite insufficient to make any definite conclusion about the benefit or otherwise of thrombolysis to treat acute ischaemic stroke.” Additional trials have since been published, and an updated version of the same review (issue 2, 1997) concluded that, although more research was needed, clear evidence existed for a substantial excess risk of intracranial haemorrhage and early death with high doses of thrombolytic drugs. Clearly stating and showing the inadequacy of existing evidence should serve as a stimulus for conducting the appropriate and necessary trials.
The Cochrane Collaboration
The dissemination of failed reviews is an important task, which is neglected by traditional journals. The examples mentioned above illustrate that this is increasingly being taken on by the Cochrane Collaboration, along with the dissemination of many other, conclusive reviews. This international group, named after Archie Cochrane, is a unique initiative in the evaluation of healthcare interventions. In his seminal book Effectiveness and Efficiency: Random Reflections on Health Services, published in 1972, Cochrane forcefully argued that the healthcare resources should be used to provide equitably those interventions that have been shown in well designed studies to be effective.31 The collaboration's effort to prepare, disseminate, and continuously update systematic reviews of controlled trials is an essential, and timely, step towards achieving this goal. The Cochrane Collaboration will have an important role in future developments in the field of systematic reviews and meta-analysis. The collaboration's working groups are addressing many of the currently unresolved issues, including, for example, the approach to observational data and data from evaluations of screening and diagnostic tests. Ways of improving the applicability of reviews, discussed below, and of strengthening the involvement of consumer representatives, are also being studied.
Clinical application of results from meta-analyses
Single large trials showing beneficial effects of treatments do influence medical practice, whereas meta-analyses of smaller studies have generally had limited impact. For example, the use of thrombolysis to reduce mortality from myocardial infarction increased only after publication of two large trials in the late 1980s,32 33 although the same reduction in mortality had already been shown in 1982 in a meta-analysis of eight smaller studies34 and again in a 1985 meta-analysis.35 The increase in the use of thrombolysis is in line with the recommendations made in authoritative reviews and textbooks. Only after publication of the first trial by the Gruppo Italiano per lo Studio della Streptochinasi nell’Infarto Miocardico was thrombolysis increasingly recommended as routine treatment after myocardial infarction.36 The 1982 meta-analysis has received only 150 citations over the 14 years since its publication in the New England Journal of Medicine (the medical journal with the highest impact factor), whereas the 1985 meta-analysis has received about 350 citations, the same as those received by a small, inconclusive trial that was published in the same year.37 The two large trials, however, have received several thousand citations over a shorter period. Clearly, meta-analyses, even when conclusive, currently receive less attention than the trials which they pool, and this is presumably reflected in a smaller degree of influence on clinical practice.
Clinicians want results from clinical research that can usefully inform their clinical practice. Perhaps meta-analyses are seen as not providing information beyond the effect of treatment on a hypothetical “average” patient. The confidence interval, often narrow in meta-analysis, reflects how certain one can be about the size of the overall effect seen in a population. Of more relevance to the clinician, however, is how certain one can be about the effect in his or her particular patient. Although the overall effect will generally provide the best available estimate, the uncertainty with respect to a particular patient will always be greater than with respect to the overall patient group. This is because, in the same way as the effect under examination varies between the component studies in the meta-analysis, the effect further varies between different patients.38
Many clinical opinion leaders simply do not trust the results from meta-analyses. This could be seen as a cautious attitude to a relatively new technique, which is justified considering the existence of misleading meta-analyses.29 39 The emergence of the “professional meta-analyst” 40 moving monthly from issue to issue, happily engaging in areas outside their domain of primary expertise, filling the pages of the medical journals, and sometimes viewed as lackeys for governmental agencies with a cost cutting agenda, has certainly not helped here. We believe that with improved methodological standards that routinely involve thorough sensitivity analyses, confidence in the results from meta-analyses will gradually grow. Although knowledge of the accumulated evidence from clinical trials should certainly provide a strong guide for practice, it is appropriate that features of the particular clinical situation should also be incorporated into the decision making process. The failure to recognise that the world is characterised as much by difference as similarity, which may be lost to those faced by numbers not patients, has on occasion led to overconfident assertions from practitioners of meta-analysis, which have understandably antagonised clinicians. Retaining a degree of humility in the face of the diversity of humanity served by medicine, and thus admitting to greater uncertainty than may be wished, will in the end prove the best way of furthering the goals of meta-analysis and the practice of evidence-based medicine to which it contributes.
In this series we outlined and illustrated the principles, strengths, and weaknesses of meta-analysis. We believe that this technique is clearly superior to the narrative approach to reviewing medical research. In addition to providing a precise estimate of the overall treatment effect in some instances, appropriate examination of heterogeneity across individual studies can produce clinically useful information with which to guide rational and cost effective treatment decisions. Both the uncritical synthesis of data from observational studies and the unconsidered synthesis of disparate results from randomised controlled trials threaten to damage the reputation of meta-analysis.
Some of the shortcomings of meta-analysis, however, are a consequence of a more general failing with respect to the dissemination of research findings. Currently this process is highly dependent on the publication of study results in peer reviewed, English-language journals. Considerations regarding publication and location biases have shown that this can result in a selected portion of all the evidence becoming available for systematic review. This is clearly unsatisfactory and can misdirect clinical practice, whether or not a formal meta-analysis is performed. Meta-analyses based on individual patient data have shown that making such data available can contribute valuable and clinically relevant information that could not be obtained from published sources. Mechanisms to facilitate such collaborative analyses and to ensure wide accessibility of results from clinical research, including results kept as “data on file” by the pharmaceutical industry, must be developed further. The technological barriers to worldwide data exchange and collaboration are tumbling down-we can only hope that the remaining barriers, rooted in customary practice, political agendas, and commercial interests, will swiftly fall too.
The department of social medicine at the University of Bristol is part of the Medical Research Council's health services research collaboration.