BMJ No 7124 Volume 316
Education and debate Saturday 3 January 1998
Meta-analysis
Bias in location and selection of studies
Matthias Egger, George Davey SmithThis is the fourth in a series of six articles examining the procedures in conducting reliable meta-analysis in medical research
| Summary points |
| Bias can be introduced in many ways into the process of locating and selecting studies for inclusion in meta-analysis |
| Studies with significant results are more likely to get published than studies without significant results, leading to publication bias |
| Among published studies, those with significant results are more likely to get published in English, more likely to be cited, and more likely to be published repeatedly, leading to English language bias, citation bias, and multiple publication bias |
| In less developed countries, studies with significant results may be more likely to get published in a journal indexed in a literature database, which can introduce database bias |
| Criteria for including studies in a meta-analysis may be influenced by knowledge of the results of the set of potential studies, leading to inclusion bias |
| The likely presence or absence of bias should be routinely examined in sensitivity analyses and funnel plots |
| Meta-analysis has received a mixed reception since
the outset. Some people have rejected what they see as exercises in
"mega-silliness,"(1) while the purveyors of a highly
distinguished series of meta-analyses of perinatal medical
care(2) have been dismissed as "an obstetrical
Baader-Meinhof gang."(3) To some clinicians objecting to
the findings of meta-analyses, "a tool has become a
weapon."(4) At the other end of the spectrum, the
application of a technique that basically consists of calculating a
weighted average has been hailed as "Newtonian,"(5)
and it has been suggested that meta-analysis has left no place for the
narrative review article.(6) The truth is likely to lie
somewhere between these extreme views. That meta-analysis holds potential problems can be illustrated by contrasting the conclusions of two meta-analyses comparing low molecular weight heparins and standard heparin in the prevention of thrombosis after surgery.(7,8) |  |
| Characteristics of two meta-analyses of low molecular weight heparins in the prevention of thrombosis after surgery | ||
|---|---|---|
| Characteristics | Leizorovicz et al(7) | Nurmohamed et al(8) |
| Years covered | 1984-91 | 1984-91 |
| No of studies included in main analyses | 39 | 23 |
| No of patients included in main analyses | 12,375 | 8,172 |
| No of trials in common in main analyses | 18 | 18 |
| Unconfounded trials only* | Yes | No |
| Language restriction | No | Yes** |
| Unpublished data | Yes | No |
| Duplicate data extraction | Yes | No |
| Stratified by trial quality | No | Yes |
| *Trials in which the only planned differences between the
treatment and control groups relate to the type of heparin treatment.
**The analysis was restricted to trials published in English, German, or French. | ||
Publication bias
The most obvious problem is that some studies never get published. If the reasons that studies remain unpublished are associated with their outcome then the result of a meta-analysis could be seriously biased. Hypothetically, with a putative treatment that has no actual effect on a disease, studies suggesting a beneficial effect might end up being published, while an equal mass of data pointing the other way might remain unpublished. In this situation a meta-analysis of the published trials would identify a spurious beneficial treatment effect. In the field of cancer chemotherapy this has indeed been shown, in a comparison of the results from studies identified in a literature search with those contained in an international trials registry(9) (box).
A demonstration of publication bias
|
| Such publication bias has been a matter of concern in education research and psychology for over 30 years.(10-12) Work on the existence and importance of publication bias in the medical literature is more recent. Five separate studies have investigated this by following up research proposals approved by ethics committees.(13-16) Of 285 studies approved by one research ethics committee between 1984 and 1987 that had been completed and analysed, 138 (48%) had been published by 1990. Studies with significant (P0.05) results were more likely to have been published than those with non-significant results.(13) A meta-analysis of all five studies shows that this is a consistent finding, with little heterogeneity between studies (fig 1). |  | |
| Fig 1: Meta-analysis of five studies examining association of significant results and publication among research proposals submitted to ethics committees. The unadjusted odds ratios were combined by using a fixed effects model |
The odds of publishing results were three times greater if the results were significant (combined odds ratio 3.0, 95% confidence intervals 2.3 to 3.9), and such publication bias was found for both clinical trials and observational studies. Interestingly, studies continue to appear in print many years after approval by the ethics committee. Stern et al (ethics committee E) found that about 85% of studies with significant results compared with 65% of studies with null results had been published after 10 years.(16) The median time to publication was 4.8 years for studies with significant results and 8.0 years for studies with null results.
The source of funding was associated with publication or non-publication independently of study results.(13,14)(16) Studies sponsored by the pharmaceutical industry were less likely to be published than those supported by the government or by voluntary organisations, with investigators citing the data management by these companies as a reason for non-publication.(13,14) This is in agreement with a review of publications of clinical trials that grouped them into those which were sponsored by the pharmaceutical industry and those supported by other means.(17) The results of 89% of published industry-supported trials favoured combination chemotherapy over monotherapy with an alkylating agent, compared with 61% of the other trials. Similar results have been reported from an overview of trials of non-steroidal anti-inflammatory drugs.(18) The implication is that the pharmaceutical industry discourages the publication of studies that it has funded which have negative findings. Finally, multicentre studies were more likely to be published than studies from a single centre.(14) However, high quality trials were not more likely to be published than trials of lower quality.(13)
Selective submission of papers rather than selective acceptance of papers by journals seems to be the dominant contributor to publication bias.(13-16) However, that selective acceptance does occur is illustrated by the "instructions to authors" section of one major journal on diabetes, which stated that "mere confirmation of known facts will be accepted only in exceptional cases; the same applies to reports of experiments and observations having no positive outcome."(19) Many authors may not submit studies with negative findings because they anticipate rejection.
Bias in location of studies
Although publication bias has long been recognised(10) and much discussed,(20,21) other factors can contribute to biased inclusion of studies in meta-analyses. Indeed, among published studies the probability of identifying relevant trials for meta-analysis is also influenced by their results. These biases have received much less consideration than publication bias, but their consequences could be equally important.
English language bias
Meta-analyses published in English language journals are often
exclusively based on trials published in English. For example, of 36
meta-analyses published in leading English language general medical
journals from 1991 to 1993, 26 had restricted their search to studies
reported in English.(22) Investigators working in a
non-English speaking country will, however, publish some of their work
in local journals.(23-25) Authors might be more likely to
report positive findings in an international, English language journal
and negative findings in a local journal. Bias could thus be introduced
in meta-analyses based exclusively on reports in
English.(22)(24)
We have examined this issue for literature published in German. All randomised controlled trials published in five German, Swiss, and Austrian general medical journals from 1985 to 1994 were identified in a manual search.(25) The Medline database was then searched for randomised controlled trials published in English during the same 10 year period.(26) Comparison of pairs of articles published by the same first author found that 63% of trials published in English had produced significant (P0.05) results, compared with 35% of trials published in German. In logistic regression analysis the odds for publication in English were 3.8 (95% confidence interval 1.3 to 11.3) times higher if the results were significant. This association was little changed when adjusted for study sample size, design, and quality. Indeed, quality scores were closely similar for English and German language reports.(26) These findings show that for publications of randomised trials from German speaking Europe, an English language bias does exist and that ignoring trials published in German is problematic. The same situation is likely in relation to other languages, particularly European languages.
Database bias
Whereas most of the major west European journals that are
published in languages other than English are indexed in Embase or
Medline, this is not the case for journals published in less developed
countries. Among the 3,000 to 4,000 journals indexed by Medline, Embase,
or the Science Citation Index, only about 2% are from the less
developed world.(27) For example, only 30 journals out of a
total of 3861 journals indexed in Medline are published in India,
despite the fact that India is the developing country with the largest
research output(28) and that its medical research is
published in English. Studies that are published in journals not
indexed in one of the major databases means that these data are
effectively hidden from reviewers and meta-analysts. A minority of
trials will be published in indexed local or international journals,
but results and other characteristics are likely to differ between
these two groups. Indeed, trials with significant results might be more
likely to be published in an indexed journal, whereas trials with null
results are published in non-indexed journals.
Citation bias
In locating studies, researchers often supplement searches of
computerised databases by contacting experts in the field and checking
the reference lists of other studies and reviews. When reference lists
are used, citation bias could have an important role. In the field of
cholesterol lowering, trials that are supportive of a beneficial effect
are cited more frequently than unsupportive trials, regardless of the
size and quality of the studies involved(29) (box). Thus the
use of reference lists would be more likely to locate supportive
studies, which could bias the findings of a meta-analysis.
Citation and selection of studies bias
|
The journals in which papers are published could also influence the ease of their location and their inclusion in meta-analyses. One influential cholesterol lowering trial, for example, was originally planned as a study with primary prevention and secondary prevention arms.(34) The results of the primary prevention component were interpreted by the investigators as being favourable, and the results were published in the New England Journal of Medicine in 1987.(35) The secondary prevention arm finished at the same time, but in this case the results were clearly unfavourable.(36) The findings from this arm were not published until 1993, in the Annals of Medicine,(36) a journal with limited circulation. The paper in the New England Journal of Medicine received more than 450 citations in the three years after publication, whereas the article in the Annals of Medicine received 17.
Multiple publication bias
Multiple publications from single studies can lead to bias in
several ways.(37) Studies with significant results are more
likely to lead to multiple publications and
presentations,(13) which makes it more likely that they will
be located and included in a meta-analysis. Furthermore, the inclusion
of duplicated data may lead to overestimation of treatment effects, as
recently shown for trials of ondansetron, a 5-HT3 receptor
antagonist that is used for prevention and treatment of postoperative
nausea and vomiting.(38) It is not always obvious that
multiple publications come from a single study, and one set of study
participants may thus be included in an analysis twice. This is a
particular problem in multicentre trials. In one of the meta-analyses
of low molecular weight heparin,(8) for example, it seems
that combined data from a multicentre trial were included, together
with a subset of the same data that had also been reported separately
from an individual centre which contributed to the
trial.(39) Indeed, it may be difficult, if not impossible,
for meta-analysts to determine whether two papers represent duplicate
publications of one trial or two separate trials, as examples exist
where two articles reporting the same trial do not share a single
common author.(37,38)(40)
Bias in provision of data
Some additional data not reported in print are often needed for
meta-analysis - for example, in the case of reports that do not provide
adequate stratification of data - but it may prove difficult to obtain
this information. Many factors could influence the willingness of
investigators to make their data available, but one element could be
the direction of the results. Such provision of data bias could also
occur in the case of unpublished trials or trials published only as
conference abstracts.
Biased inclusion criteria
Although studies might have been located and data obtained, potential for bias might still arise in establishing the inclusion criteria for a meta-analysis. If, as is usual, the inclusion criteria are developed by an investigator familiar with the area under study, the criteria can be influenced by knowledge of the results of the set of potential studies. Manipulating the inclusion criteria could lead to selective inclusion of studies with positive findings and exclusion of studies with negative findings. For example, some meta-analyses of trials of cholesterol lowering treatment have excluded certain studies on the grounds that the treatments used seem to have had an adverse effect independent of cholesterol lowering itself.(41)(42) These meta-analyses have, however, included trials of treatments that are likely to influence favourably the risk of coronary heart disease, independent of cholesterol lowering. Clearly such an asymmetrical approach introduces the possibility of selection bias, with the criteria for inclusion in the meta-analysis being derived from the results of the studies (see box on citation and selection bias).
A more recent example relates to a meta-analysis of trials of dietary interventions in community settings.(43) The authors excluded a trial because the randomisation process was considered to be inadequate, but they included a duplicate publication of the same trial with an almost identical description of the randomisation procedure.(44) Two independent observers assessed eligibility of trials, with disagreements being resolved by a third author. This example shows how vulnerable the selection process can be in meta-analysis.
Examining for bias
The most appropriate way of handling the selection of studies is to include all studies that meet basic entry criteria then perform sensitivity analyses with regard to the different possible entry criteria. Any conclusions from a meta-analysis that are highly sensitive to altering the entry criteria should be treated with caution. In addition to such sensitivity analyses, the likely presence or absence of bias should be examined graphically in funnel plots.
Funnel plots
Funnel plots - simple scatterplots of the trials' effect estimates against their sample size - are useful to detect bias in meta-analysis.(45) The funnel plot is based on the fact that precision in estimating the underlying treatment effect will increase as the sample size of component studies increases. Results from small studies will scatter widely at the bottom of the graph. The spread will narrow as precision increases among larger studies. In the absence of bias, the plot should thus resemble a symmetrical inverted funnel. If the plot shows an asymmetrical and skewed shape, bias may be present.
This usually takes the form of a gap in the wide part of the funnel, which indicates the absence of small studies showing no benefit or harm. The funnel plot is a graphical test for any type of bias that is associated with sample size (box). The publication and location biases described in this article are more likely to affect smaller studies than large trials and may thus lead to funnel plot asymmetry. Another source of asymmetry arises from differences in the methodological quality. Smaller studies are, on average, conducted and analysed with less methodological rigour than larger studies, and trials of lower quality tend to show larger effects.(46) Statistical methods that provide an objective measure of funnel plot asymmetry have recently become available.(45)(47)
Biases in meta-analysis that may be detected in funnel plots
|
| In this series we have repeatedly discussed a meta-analysis of 17 trials assessing ß blockers in secondary prevention after myocardial infarction. The sensitivity analysis that we presented earlier(48) showed that the results were robust to the choice of the statistical method and to the exclusion of trials of lesser quality or of studies terminated early. The funnel plot is shown in figure 2. Visual assessment shows some asymmetry, which indicates that there was selective non-publication of smaller trials with less sizeable benefit. However, in formal statistical analysis(45)(47) the degree of asymmetry is found to be small and non-significant (P>0.1). Furthermore, exclusion of the smaller studies had little effect on the overall estimate.(48) Bias does not therefore seem to have distorted the findings from this meta-analysis. |  | |
| Fig 2: Funnel plot of mortality results from trials of ß blockers in secondary prevention after myocardial infarction. The odds ratios are plotted against study sample size |
Conclusions
Biases in publication, location, and inclusion are a potentially serious problem in meta-analysis(49-51) Critical examination for the presence of such biases in sensitivity and funnel plot analyses should therefore form an integral part of meta-analyses.(45) The effort of the Cochrane Collaboration to identify as many controlled trials as possible through manual searches of a large number of medical journals published in many different languages is of great importance to reduce such bias. Indeed, the Cochrane Controlled Trials Register(52) is nowadays likely to be the best single source of published trials for inclusion in systematic reviews and meta-analyses. To eliminate the risk of publication bias, however, trials need to be registered at the time they are established. To ensure registration, the ethical approval of studies could be linked to a requirement that trials are reported to a central register.(33)(49)(53) Furthermore, results of completed trials could be submitted to the ethical committee, the reports could be kept centrally, and requests for unpublished trials could be sent to this body. At present, however, much effort is still needed to ensure that the set of studies located for a meta-analysis is not a biased sample of all existing studies.
The department of social medicine at the University of Bristol is part of the Medical Research Council's health services research collaboration.
Funding: ME was supported by the Swiss National Science Foundation.
Department of Social Medicine,
University of Bristol,
Bristol BS8 2PR
Matthias
Egger,
reader in social medicine and
epidemiology
George Davey Smith, professor of
clinical epidemiology
Correspondence to: Dr Egger
email: m.egger@bristol.ac.uk
References
1 Eysenck H J. An exercise in mega-silliness. Am
Psychol 1978;33:517.
2 Chalmers I, Enkin M, Keirse M. Effective care during
pregnancy and childbirth. Oxford: Oxford University Press,
1989.
3 Mann C. Meta-analysis in the breech. Science
1990;249:476-80.
4 Boden W E. Meta-analysis in clinical trials reporting: has a
tool become a weapon? Am J Cardiol 1992;69:681-6.
5 Mann C, Peto R. Statistician with a mission.
Science 1990;249:479.
6 Chalmers T C, Frank C S, Reitman D. Minimizing the three stages
of publication bias. JAMA 1990;263:1392-5.
7 Leizorovicz A, Haugh M C, Chapuis F-R, Samama M M, Boissel J-P.
Low molecular weight heparin in prevention of perioperative thrombosis.
BMJ 1992;305:913-20.
8 Nurmohamed M T, Rosendaal F R, Bueller H R, Dekker E, Hommes D W,
Vandenbroucke J P, et al. Low-molecular-weight heparin versus standard
heparin in general and orthopaedic surgery: a meta-analysis.
Lancet 1992;340:152-6.
9 Simes R J. Confronting publication bias: a cohort design for
meta-analysis. Stat Med 1987;6:11-29.
10 Sterling T D. Publication decisions and their possible effects
on inferences drawn from tests of significance - or vice versa. J
Am Stat Assoc 1959;54:30-4.
11 Rosenthal R. The "file drawer problem" and tolerance for
null results. Psychol Bull 1979;86:638-41.
12 Light R J, Pillemer D B. Summing up. The science of
reviewing research. Cambridge, MA; London: Harvard University
Press, 1984.
13 Easterbrook P J, Berlin J A, Gopalan R, Matthews D R. Publication
bias in clinical research. Lancet 1991;337:867-72.
14 Dickersin K, Min Y L, Meinert C L. Factors influencing
publication of research results. Follow-up of applications submitted to
two institutional review boards. JAMA 1992;267:374-8.
15 Dickersin K, Min Y L. NIH clinical trials and publication bias.
Online J Curr Clin Trials [serial on line] 1993;28
Apr:Doc No 50.
16 Stern J M, Simes R J. Publication bias: evidence of delayed
publication in a cohort study of clinical research projects.
BMJ 1997;315:640-5.
17 Davidson R A. Source of funding and outcome of clinical trials.
J Gen Intern Med 1986;1:155-8.
18 Rochon P A, Gurwitz J H, Simms R W, Fortin P R, Felson D T, Minaker
K L, et al. A study of manufacturer-supported trials of nonsteroidal
anti-inflammatory drugs in the treatment of arthritis. Arch
Intern Med 1994;154:157-63.
19 Manuscript guideline. Diabetologia 1984;25:4A.
20 Dickersin K, Chan S, Chalmers T C, Sacks H S, Smith H.
Publication bias in clinical trials. Controlled Clinical Trials
1987;8:343-53.
21 Begg C B, Berlin J A. Publication bias and dissemination of
clinical research. J Natl Cancer Inst 1989;81:107-15.
22 Grégoire G, Derderian F, LeLorier J. Selecting the language
of the publications included in a meta-analysis: is there a Tower of
Babel bias? J Clin Epidemiol 1995;48:159-63.
23 Dickersin K, Scherer R, Lefebvre C. Identifying relevant
studies for systematic reviews. BMJ 1994;309:1286-91.
24 Moher D, Fortin P, Jadad A R, Jni P, Klassen T, Le Lorier J,
et al. Completeness of reporting of trials published in languages other
than English: implications for conduct and reporting of systematic
reviews. Lancet 1996;347:363-6.
25 Egger M, Zellweger T, Antes G. Randomised trials in
German-language journals. Lancet 1996;347:1047-8.
26 Egger M, Zellweger-Zóhner T, Schneider M, Junker C, Lengeler
C, Antes G. Language bias in randomised controlled trials published in
English and German. Lancet 1997;350:326-9.
27 Zielinski C. New equities of information in an electronic age.
BMJ 1995;1480-1.
28 Singh R, Singh S. Research and doctors. Lancet
1994;344:546.
29 Ravnskov U. Cholesterol lowering trials in coronary heart
disease: frequency of citation and outcome. BMJ
1992;305:15-9.
30 Rossouw J E, Lewis B, Rifkind B M. The value of lowering
cholesterol after myocardial infarction. N Engl J Med
1990;323:1112-9.
31 Woodhill J M, Palmer A J, Leelarthaepin B, McGilchrist C, Blacket
R B. Low fat, low cholesterol diet in secondary prevention of coronary
heart disease. Adv Exp Med Biol 1978;109:317-30.
32 Davey Smith G, Song F, Sheldon T A. Cholesterol lowering and
mortality: the importance of considering initial level of risk.
BMJ 1993;306:1367-73.
33 Savulescu J, Chalmers I, Blunt J. Are research ethics
committees behaving unethically? Some suggestions for improving
performances and accountability. BMJ 1996;313:1390-3.
34 Manninen V. Clinical results with gefibrozil and background to
the Helsinki heart study. Am J Cardiol 1983;52:35-8B.
35 Frick M H, Elo O, Haapa K, Heinonen O P, Heinsalmi P, Helo P, et
al. Helsinki heart study: primary-prevention trial with gemfibrozil in
middle-aged men with dyslipidemia. N Engl J Med
1987;317:1237-45.
36 Frick M H, Heinonen O P, Huttunen J K, Koskinen P, Manttari M,
Manninen V. Efficacy of gemfibrozil in dyslipidaemic subjects with
suspected heart disease. An ancillary study in the Helsinki heart study
frame population. Ann Med 1993;25:41-5.
37 Huston P, Moher D. Redundancy, disaggregation, and the
integrity of medical research. Lancet 1996;347:1024-6.
38 Tramér M R, Reynolds D J M, Moore R A, McQuay H J. Impact of
covert duplicate publication on meta-analysis: a case study.
BMJ 1997;315:635-40.
39 Leizorovicz A, Haugh M C, Boissel J P. Meta-analysis and multiple
publications of clinical trial reports. Lancet
1992;340:1102-3.
40 Felson D T. Bias in meta-analytic research. J Clin
Epidemiol 1992;45:885-92.
41 Peto R, Yusuf S, Collins R. Cholesterol-lowering trial results
in their epidemiologic context. Circulation
1985;72(suppl 3):451.
42 MacMahon S. Lowering cholesterol: effects on trauma death,
cancer death and total mortality. Aust N Z J Med
1992;22:580-2.
43 Tang J L, Armitage J M, Lancaster T, Silagy C A, Fowler G H, Neil
H A W. Systematic review of dietary intervention trials to lower
cholesterol in free-living individuals. BMJ (in press).
44 Davey Smith G, Ebrahim S. Dietary change, cholesterol reduction
and the public health: what does meta-analysis add? BMJ
(in press).
45 Egger M, Davey Smith G, Schneider M, Minder C E. Bias in
meta-analysis detected by a simple, graphical test. BMJ
1997;315:629-34.
46 Schulz K F, Chalmers I, Hayes R J, Altman D G. Empirical evidence
of bias. Dimensions of methodological quality associated with estimates
of treatment effects in controlled trials. JAMA
1995;273:408-12.
47 Begg C B, Mazumdar M. Operating characteristics of a rank
correlation test for publication bias. Biometrics
1994;50:1088-99.
48 Egger M, Davey Smith G, Phillips A N. Meta-analysis: principles
and procedures. BMJ 1997;315:1533-7.
49 Chalmers I. Underreporting research is scientific misconduct.
JAMA 1990;263:1405-8.
50 Egger M, Davey Smith G. Misleading meta-analysis. Lessons from
"an effective, safe, simple" intervention that wasn't.
BMJ 1995;310:752-4.
51 Pearn J. Publication: an ethical imperative. BMJ
1995;310:1313-5.
52 Cochrane Controlled Trials Register. In: The Cochrane
Library [database on disk and CD ROM]. The Cochrane
Collaboration. Issue 1. Oxford: Update Software, 1997. Updated
quarterly.
53 Levy G. Publication bias: its implications for clinical
pharmacology. Clin Pharmacol Ther
1992;52:115-9.
Home | Current issue | Past issues | Classified ads | Career Focus | Feedback
Collections | About this site | About the BMJ | BMA | Medline