meta-analysis bias in location and selection of studies
(Published 03 January 1998)
Cite this as: BMJ 1998;316:61
- Matthias Egger, reader in social medicine and epidemiologya (, )
- George Davey Smith, professor of clinical epidemiologya
- Correspondence to: Dr Egger
Meta-analysis has received a mixed reception since the outset. Some people have rejected what they see as exercises in “mega-silliness,”1 while the purveyors of a highly distinguished series of meta-analyses of perinatal medical care2 have been dismissed as “an obstetrical Baader-Meinhof gang.”3 To some clinicians objecting to the findings of meta-analyses, “a tool has become a weapon.”4 At the other end of the spectrum, the application of a technique that basically consists of calculating a weighted average has been hailed as “Newtonian,”5 and it has been suggested that meta-analysis has left no place for the narrative review article.6 The truth is likely to lie somewhere between these extreme views.
Bias can be introduced in many ways into the process of locating and selecting studies for inclusion in meta-analysis
Studies with significant results are more likely to get published than studies without significant results, leading to publication bias
Among published studies, those with significant results are more likely to get published in English, more likely to be cited, and more likely to be published repeatedly, leading to English language bias, citation bias, and multiple publication bias
In less developed countries, studies with significant results may be more likely to get published in a journal indexed in a literature database, which can introduce database bias
Criteria for including studies in a meta-analysis may be influenced by knowledge of the results of the set of potential studies, leading to inclusion bias
The likely presence or absence of bias should be routinely examined in sensitivity analyses and funnel plots
That meta-analysis holds potential problems can be illustrated by contrasting the conclusions of two meta-analyses comparing low molecular weight heparins and standard heparin in the prevention of thrombosis after surgery.7 8 One group concluded that “low molecular weight heparins seem to have a higher benefit to risk ratio than unfractionated heparin in preventing perioperative thrombosis,”7 whereas the other considered that “there is at present no convincing evidence that in general surgery patients low molecular weight heparins, compared with standard heparin, generate a clinically important improvement in the benefit to risk ratio.”8 Various differences exist between these meta-analyses, but the main difference relates to the selection of studies for inclusion (1). Nurmohamed et al based their analysis on a subgroup of trials that they considered to possess the highest methodological strength,8 while Leizorovicz et al included all trials in their analysis.7 The 1 shows that many other elements—for example, language restrictions or use of unpublished material—could contribute to contrasting conclusions being reached.
The most obvious problem is that some studies never get published. If the reasons that studies remain unpublished are associated with their outcome then the result of a meta-analysis could be seriously biased. Hypothetically, with a putative treatment that has no actual effect on a disease, studies suggesting a beneficial effect might end up being published, while an equal mass of data pointing the other way might remain unpublished. In this situation a meta-analysis of the published trials would identify a spurious beneficial treatment effect. In the field of cancer chemotherapy this has indeed been shown, in a comparison of the results from studies identified in a literature search with those contained in an international trials registry9 (box).
A demonstration of publication bias
Studies with significant results are more likely to get published than those with non-significant results
The inclusion of a study in a trials register can be assumed not to be influenced by its results because registration generally takes place before completion of the study
The studies enlisted in a register are therefore likely to constitute a more representative sample of all the studies that have been performed in a given area than a sample of published studies
This principle has been tested for trials of different cancer chemotherapies by comparing the results from meta-analysis of trials identified in a literature search and of trials registered with the International Cancer Research Data Bank (4)9
Such publication bias has been a matter of concern in education research and psychology for over 30 years.10 11 12 Work on the existence and importance of publication bias in the medical literature is more recent. Five separate studies have investigated this by following up research proposals approved by ethics committees.13 14 15 16 Of 285 studies approved by one research ethics committee between 1984 and 1987 that had been completed and analysed, 138 (48%) had been published by 1990. Studies with significant (P<0.05) results were more likely to have been published than those with non-significant results.13 A meta-analysis of all five studies shows that this is a consistent finding, with little heterogeneity between studies (fig 1). The odds of publishing results were three times greater if the results were significant (combined odds ratio 3.0, 95% confidence intervals 2.3 to 3.9), and such publication bias was found for both clinical trials and observational studies. Interestingly, studies continue to appear in print many years after approval by the ethics committee. Stern et al (ethics committee E) found that about 85% of studies with significant results compared with 65% of studies with null results had been published after 10 years.16 The median time to publication was 4.8 years for studies with significant results and 8.0 years for studies with null results.
The source of funding was associated with publication or non-publication independently of study results.13 14 16 Studies sponsored by the pharmaceutical industry were less likely to be published than those supported by the government or by voluntary organisations, with investigators citing the data management by these companies as a reason for non-publication.13 14 This is in agreement with a review of publications of clinical trials that grouped them into those which were sponsored by the pharmaceutical industry and those supported by other means.17 The results of 89% of published industry-supported trials favoured combination chemotherapy over monotherapy with an alkylating agent, compared with 61% of the other trials. Similar results have been reported from an overview of trials of non-steroidal anti-inflammatory drugs.18 The implication is that the pharmaceutical industry discourages the publication of studies that it has funded which have negative findings. Finally, multicentre studies were more likely to be published than studies from a single centre.14 However, high quality trials were not more likely to be published than trials of lower quality.13
Selective submission of papers rather than selective acceptance of papers by journals seems to be the dominant contributor to publication bias.13 14 15 16 However, that selective acceptance does occur is illustrated by the “instructions to authors” section of one major journal on diabetes, which stated that “mere confirmation of known facts will be accepted only in exceptional cases; the same applies to reports of experiments and observations having no positive outcome.”19 Many authors may not submit studies with negative findings because they anticipate rejection.
Bias in location of studies
Although publication bias has long been recognised10 and much discussed,20 21 other factors can contribute to biased inclusion of studies in meta-analyses. Indeed, among published studies the probability of identifying relevant trials for meta-analysis is also influenced by their results. These biases have received much less consideration than publication bias, but their consequences could be equally important.
English language bias
Meta-analyses published in English language journals are often exclusively based on trials published in English. For example, of 36 meta-analyses published in leading English language general medical journals from 1991 to 1993, 26 had restricted their search to studies reported in English.22 Investigators working in a non-English speaking country will, however, publish some of their work in local journals.23 24 25 Authors might be more likely to report positive findings in an international, English language journal and negative findings in a local journal. Bias could thus be introduced in meta-analyses based exclusively on reports in English.22 24
We have examined this issue for literature published in German. All randomised controlled trials published in five German, Swiss, and Austrian general medical journals from 1985 to 1994 were identified in a manual search.25 The Medline database was then searched for randomised controlled trials published in English during the same 10 year period.26 Comparison of pairs of articles published by the same first author found that 63% of trials published in English had produced significant (P<0.05) results, compared with 35% of trials published in German. In logistic regression analysis the odds for publication in English were 3.8 (95% confidence interval 1.3 to 11.3) times higher if the results were significant. This association was little changed when adjusted for study sample size, design, and quality. Indeed, quality scores were closely similar for English and German language reports.26 These findings show that for publications of randomised trials from German speaking Europe, an English language bias does exist and that ignoring trials published in German is problematic. The same situation is likely in relation to other languages, particularly European languages.
Whereas most of the major west European journals that are published in languages other than English are indexed in Embase or Medline, this is not the case for journals published in less developed countries. Among the 3000 to 4000 journals indexed by Medline, Embase, or the Science Citation Index, only about 2% are from the less developed world.27 For example, only 30 journals out of a total of 3861 journals indexed in Medline are published in India, despite the fact that India is the developing country with the largest research output28 and that its medical research is published in English. Studies that are published in journals not indexed in one of the major databases means that these data are effectively hidden from reviewers and meta-analysts. A minority of trials will be published in indexed local or international journals, but results and other characteristics are likely to differ between these two groups. Indeed, trials with significant results might be more likely to be published in an indexed journal, whereas trials with null results are published in non-indexed journals.
In locating studies, researchers often supplement searches of computerised databases by contacting experts in the field and checking the reference lists of other studies and reviews. When reference lists are used, citation bias could have an important role. In the field of cholesterol lowering, trials that are supportive of a beneficial effect are cited more frequently than unsupportive trials, regardless of the size and quality of the studies involved29 (box). Thus the use of reference lists would be more likely to locate supportive studies, which could bias the findings of a meta-analysis.
Citation and selection of studies bias
A meta-analysis of seven trials of cholesterol lowering after myocardial infarction30 defined its inclusion criteria as single-factor randomised trials with at least 100 participants per group, with at least three years of follow up, and without the use of hormone treatment to lower cholesterol. The pooled results for all cause mortality showed a favourable trend (odds ratio 0.91, 95% confidence interval 0.82 to 1.02) (5).
One trial met all the entry criteria but was not included.31 In this study the odds ratio for overall mortality was an unfavourable 1.60 (0.95 to 2.70). For the trials included in the analysis the mean annual citation count per study for the period up to five years after publication was 20; for the study that was not included it was less than 1.29 Citation bias may thus have distorted the results of this meta-analysis.
Eleven other secondary prevention trials were available at the time but did not meet the somewhat arbitrary inclusion criteria.32 The pooled odds ratio for all cause mortality for these trials is 1.14 (1.03 to 1.26). Inclusion biases may thus have influenced the conclusions of this meta-analysis.
The journals in which papers are published could also influence the ease of their location and their inclusion in meta-analyses. One influential cholesterol lowering trial, for example, was originally planned as a study with primary prevention and secondary prevention arms.34 The results of the primary prevention component were interpreted by the investigators as being favourable, and the results were published in the New England Journal of Medicine in 1987.35 The secondary prevention arm finished at the same time, but in this case the results were clearly unfavourable.36 The findings from this arm were not published until 1993, in the Annals of Medicine,36 a journal with limited circulation. The paper in the New England Journal of Medicine received more than 450 citations in the three years after publication, whereas the article in the Annals of Medicine received 17.
Multiple publication bias
Multiple publications from single studies can lead to bias in several ways.37 Studies with significant results are more likely to lead to multiple publications and presentations,13 which makes it more likely that they will be located and included in a meta-analysis. Furthermore, the inclusion of duplicated data may lead to overestimation of treatment effects, as recently shown for trials of ondansetron, a 5-HT3 receptor antagonist that is used for prevention and treatment of postoperative nausea and vomiting.38 It is not always obvious that multiple publications come from a single study, and one set of study participants may thus be included in an analysis twice. This is a particular problem in multicentre trials. In one of the meta-analyses of low molecular weight heparin,8 for example, it seems that combined data from a multicentre trial were included, together with a subset of the same data that had also been reported separately from an individual centre which contributed to the trial.39 Indeed, it may be difficult, if not impossible, for meta-analysts to determine whether two papers represent duplicate publications of one trial or two separate trials, as examples exist where two articles reporting the same trial do not share a single common author.37 38 40
Bias in provision of data
Some additional data not reported in print are often needed for meta-analysis—for example, in the case of reports that do not provide adequate stratification of data—but it may prove difficult to obtain this information. Many factors could influence the willingness of investigators to make their data available, but one element could be the direction of the results. Such provision of data bias could also occur in the case of unpublished trials or trials published only as conference abstracts.
Biased inclusion criteria
Although studies might have been located and data obtained, potential for bias might still arise in establishing the inclusion criteria for a meta-analysis. If, as is usual, the inclusion criteria are developed by an investigator familiar with the area under study, the criteria can be influenced by knowledge of the results of the set of potential studies. Manipulating the inclusion criteria could lead to selective inclusion of studies with positive findings and exclusion of studies with negative findings. For example, some meta-analyses of trials of cholesterol lowering treatment have excluded certain studies on the grounds that the treatments used seem to have had an adverse effect independent of cholesterol lowering itself.41 42 These meta-analyses have, however, included trials of treatments that are likely to influence favourably the risk of coronary heart disease, independent of cholesterol lowering. Clearly such an asymmetrical approach introduces the possibility of selection bias, with the criteria for inclusion in the meta-analysis being derived from the results of the studies (see box on citation and selection bias).
A more recent example relates to a meta-analysis of trials of dietary interventions in community settings.43 The authors excluded a trial because the randomisation process was considered to be inadequate, but they included a duplicate publication of the same trial with an almost identical description of the randomisation procedure.44 Two independent observers assessed eligibility of trials, with disagreements being resolved by a third author. This example shows how vulnerable the selection process can be in meta-analysis.
Examining for bias
The most appropriate way of handling the selection of studies is to include all studies that meet basic entry criteria then perform sensitivity analyses with regard to the different possible entry criteria. Any conclusions from a meta-analysis that are highly sensitive to altering the entry criteria should be treated with caution. In addition to such sensitivity analyses, the likely presence or absence of bias should be examined graphically in funnel plots.
Funnel plots—simple scatterplots of the trials' effect estimates against their sample size—are useful to detect bias in meta-analysis.45 The funnel plot is based on the fact that precision in estimating the underlying treatment effect will increase as the sample size of component studies increases. Results from small studies will scatter widely at the bottom of the graph. The spread will narrow as precision increases among larger studies. In the absence of bias, the plot should thus resemble a symmetrical inverted funnel. If the plot shows an asymmetrical and skewed shape, bias may be present. This usually takes the form of a gap in the wide part of the funnel, which indicates the absence of small studies showing no benefit or harm. The funnel plot is a graphical test for any type of bias that is associated with sample size (box). The publication and location biases described in this article are more likely to affect smaller studies than large trials and may thus lead to funnel plot asymmetry. Another source of asymmetry arises from differences in the methodological quality. Smaller studies are, on average, conducted and analysed with less methodological rigour than larger studies, and trials of lower quality tend to show larger effects.46 Statistical methods that provide an objective measure of funnel plot asymmetry have recently become available.45 47
Biases in meta-analysis that may be detected in funnel plots
English language bias
Multiple publication bias
Bias in provision of data
Poor methodological quality of small studies
An example—ß blockers after myocardial infarction
In this series we have repeatedly discussed a meta-analysis of 17 trials assessing β blockers in secondary prevention after myocardial infarction. The sensitivity analysis that we presented earlier48 showed that the results were robust to the choice of the statistical method and to the exclusion of trials of lesser quality or of studies terminated early. The funnel plot is shown in figure 3. Visual assessment shows some asymmetry, which indicates that there was selective non-publication of smaller trials with less sizeable benefit. However, in formal statistical analysis45 47 the degree of asymmetry is found to be small and non-significant (P>0.1). Furthermore, exclusion of the smaller studies had little effect on the overall estimate.48 Bias does not therefore seem to have distorted the findings from this meta-analysis.
Biases in publication, location, and inclusion are a potentially serious problem in meta-analysis49 50 51 Critical examination for the presence of such biases in sensitivity and funnel plot analyses should therefore form an integral part of meta-analyses.45 The effort of the Cochrane Collaboration to identify as many controlled trials as possible through manual searches of a large number of medical journals published in many different languages is of great importance to reduce such bias. Indeed, the Cochrane Controlled Trials Register52 is nowadays likely to be the best single source of published trials for inclusion in systematic reviews and meta-analyses. To eliminate the risk of publication bias, however, trials need to be registered at the time they are established. To ensure registration, the ethical approval of studies could be linked to a requirement that trials are reported to a central register.33 49 53 Furthermore, results of completed trials could be submitted to the ethical committee, the reports could be kept centrally, and requests for unpublished trials could be sent to this body. At present, however, much effort is still needed to ensure that the set of studies located for a meta-analysis is not a biased sample of all existing studies.
The department of social medicine at the University of Bristol is part of the Medical Research Council's health services research collaboration.