BMJ 2001;323:42-46 ( 7 July )

Education and debate

Systematic reviews in health care

Assessing the quality of controlled clinical trials

This is the first in a series of four articles

Peter Jüni, research fellow a, Douglas G Altman, professor of statistics in medicine b, Matthias Egger, senior lecturer in epidemiology and public health medicine c. 

a Department of Social and Preventive Medicine, University of Bern, Bern, 3012 Switzerland, b Imperial Cancer Research Fund Medical Statistics Group, Centre for Statistics in Medicine, Institute of Health Sciences, Oxford OX3 7LF, c Medical Research Council Health Services Research Collaboration, Department of Social Medicine, University of Bristol, Bristol BS8 2PR

Correspondence to: M Egger m.egger{at}bristol.ac.uk

The quality of controlled trials is of obvious relevance to systematic reviews. If the "raw material" is flawed then the conclusions of systematic reviews cannot be trusted. Many reviewers formally assess the quality of primary trials by following the recommendations of the Cochrane Collaboration and other experts. 1 2 However, the methodology for both the assessment of quality and its incorporation into systematic reviews and meta-analysis are a matter of ongoing debate.3-5 In this article we discuss the concept of study quality and the methods used to assess quality.


Components of internal and external validity of controlled clinical trials

Internal validity---extent to which systematic error (bias) is minimised in clinical trials

  • Selection bias: biased allocation to comparison groups

  • Performance bias: unequal provision of care apart from treatment under evaluation

  • Detection bias: biased assessment of outcome

  • Attrition bias: biased occurrence and handling of deviations from protocol and loss to follow up
  • External validity---extent to which results of trials provide a correct basis for generalisation to other circumstances

  • Patients: age, sex, severity of disease and risk factors, comorbidity
  • Treatment regimens: dosage, timing and route of administration, type of treatment within a class of treatments, concomitant treatments
  • Settings: level of care (primary to tertiary) and experience and specialisation of care provider
  • Modalities of outcomes: type or definition of outcomes and duration of follow up

Quality is a multidimensional concept, which could relate to the design, conduct, and analysis of a trial, its clinical relevance, or quality of reporting.6 The validity of the findings generated by a study clearly is an important dimension of quality. In the 1950s the social scientist Campbell proposed a useful distinction between internal and external validity (see box below). 7 8 Internal validity implies that the differences observed between groups of patients allocated to different interventions may, apart from random error, be attributed to the treatment under investigation. In contrast, external validity, or generalisability, is the extent to which the results of a study provide a correct basis for generalisations to other circumstances. In itself, there is no external validity. The term is only meaningful with regard to specified "external" conditions, such as other patient populations or treatment regimens. Internal validity is a prerequisite for external validity: the results of a flawed trial are invalid, and the question of its external validity becomes redundant.


Summary points

Empirical studies show that inadequate quality of trials may distort the results from systematic reviews and meta-analyses
The influence of the quality of included studies should routinely be examined in systematic reviews and meta-analyses
The use of summary scores from quality scales is problematic---it is preferable to examine the influence of key components of methodological quality individually
Based on empirical evidence and theoretical considerations, the generation and concealment of the allocation sequence, blinding, and handling of patient attrition in the analysis should always be assessed




    Dimensions of internal validity

Internal validity is threatened by bias, "any process at any stage of inference tending to produce results that differ systematically from the true values."9 In clinical trials, biases fall into four categories: selection bias, performance bias, detection bias, and attrition bias (box).

Selection bias
The aim of randomisation is the creation of groups that are comparable for any known or unknown potential confounding factors.10 Success depends on two interrelated procedures (see box above).11 Firstly, an allocation sequence that is suitable to prevent selection bias must be generated--- for example, by using a computer algorithm, tossing a coin, or throwing a dice. Secondly, this sequence must be concealed from investigators enrolling patients. Knowledge of assignments---for example, from a table of random numbers posted on a bulletin board---can cause selective enrolment of patients on the basis of prognostic factors.12 Patients who would have been assigned to a treatment deemed to be "inappropriate" may be rejected, and some patients may be deliberately directed to the "appropriate" treatment.13 Deciphering of allocation schedules may occur even if concealment was attempted. For example, envelopes may be opened or held against a bright light to reveal the contents.14


The two interrelated steps of randomisation

Generation of allocation sequences

  • Adequate if sequences are suitable to prevent selection bias: random numbers generated by computer, table of random numbers, drawing of lots or envelopes, tossing a coin, shuffling cards, throwing dice, etc

  • Inadequate if sequences could be related to prognosis and thus introduce selection bias: case record number; date of birth; day, month, or year of admission; etc
  • Concealment of allocation sequences

  • Adequate if patients and investigators enrolling patients cannot foresee assignment: a priori numbered or coded drug containers of identical appearance prepared by an independent pharmacy; central randomisation (performed at a site remote from the trial's location); sequentially numbered, sealed, opaque envelopes; etc

  • Inadequate if patients and investigators enrolling patients can foresee assignments and thus introduce selection bias: procedures based on inadequate generation of allocation sequences, open allocation schedule, alternation and other unsealed or non-opaque envelopes, etc

Performance bias and detection bias
Performance bias occurs if additional treatment interventions are provided preferentially to one group. Blinding of patients and care providers prevents this type of bias and also safeguards against differences in placebo responses between the groups. Detection bias arises if the knowledge of patient assignment influences the assessment of outcome.15 This is avoided by the blinding of those assessing outcomes---for example, patients, care providers, radiologists, or end point review committees (box).

Attrition bias
Deviations from protocol and loss to follow up often lead to the exclusion of patients after they have been allocated to treatment groups, which may introduce attrition bias. Possible deviations from protocol include the violation of eligibility criteria and non-adherence to treatments. Loss to follow up refers to patients becoming unavailable for examinations at some stage during the study period because they refuse to participate further (also called drop outs), cannot be contacted, or clinical decisions are made to stop the assigned interventions.

Patients excluded after allocation are unlikely to be representative of patients remaining in the study. For example, patients may not be available for follow up because they have an acute exacerbation of their illness or severe side effects.16 Patients not adhering to treatments generally differ in respects that are related to prognosis.17 All randomised patients should therefore be included in the analysis and kept in their original groups, regardless of their adherence to the study protocol. In other words the analysis should be performed according to the intention to treat principle, thus avoiding selection bias. 16 18 This implies that the primary outcome was recorded for all randomised patients at the prespecified times throughout the follow up period.19 If the end point of interest is mortality from all causes this can be established most of the time. It may, however, be impossible retrospectively to ascertain other binary or continuous outcomes, and some patients may therefore have to be excluded from the analysis. In this case the proportion of patients not included in the analysis must be reported and the possibility of attrition bias discussed.


    Empirical evidence of bias

Numerous case studies show that the biases described above do occur in practice, distorting the results of clinical trials.6 The authors are aware of four methodological studies that have gauged their relative importance in a large number of clinical trials while avoiding confounding by disease or intervention.20-23 The figure shows a meta-analysis of the results from these studies. Inadequate or unclear concealment of treatment allocation was associated with an exaggeration of treatment effects in all four studies. Odds ratios from trials with inadequate or unclear concealment were on average 30% lower (more beneficial) than those from trials with adequate methodology (combined ratio of odds ratios 0.70, 95% confidence interval 0.62 to 0.80). The inappropriate generation of allocation sequences was assessed in three studies only and was not consistently associated with treatment effects, although an effect was evident in the study from Denmark (figure). 20 21 23 Interestingly, when only trials with adequate concealment of allocation were analysed in Schulz et al's study, those with an inadequate generation of allocation sequences did yield inflated treatment effects.20 This indicates that if assignments are predictable some deciphering can occur, even with adequate concealment. On the other hand, the generation of unbiased sequences is probably irrelevant if the sequences are not concealed from those involved in the enrolment of patients.13



View larger version (32K):
[in this window]
[in a new window]
  		Meta-analysis of four empirical studies relating key aspects of methodological quality of controlled trials to their effect estimates. Meta-analysis was by random effects model. Size of squares is proportional to inverse of variance of estimate

Results for double blinding were more heterogeneous: the two larger studies 20 22 found that estimates were on average moderately biased in open trials, whereas one of the two smaller studies showed no effect,21 and the other showed substantial bias associated with lack of double blinding (figure).23 To some extent the importance of blinding depends on the outcomes assessed. In some situations---for example, when examining the effect of an intervention on overall mortality---blinding of outcome assessment is irrelevant. Differences in the type of outcomes examined could thus explain the discrepancy between the studies.

Furthermore, investigators' understanding of who exactly should be blinded in double blind trials varies,24 and this may also introduce heterogeneity. Two studies addressed attrition bias but used different definitions. Schulz et al compared trials that reported exclusions with trials that either explicitly reported no exclusions or gave the impression that no exclusions had taken place.20 In contrast, Kjaergard et al compared trials that reported adequately on attrition (independent of whether exclusions occurred) to trials with inadequate reporting.23 Schulz et al found little difference in effect estimates (ratio of odds ratios 1.07, 95% confidence interval 0.94 to 1.21) whereas Kjaergard et al found a trend towards larger effect estimates in trials with adequate reporting (ratio of odds ratios 1.50, 0.80 to 2.78). 20 23 The methods used to assess attrition were unsatisfactory in both of these studies. Future research in this area should distinguish between quality of reporting and methodological quality and consider that some exclusions and losses to follow up may be unavoidable whereas others are clearly inappropriate.


    Dimensions of external validity

External validity relates to the applicability of the results of a study to other "populations, settings, treatment variables, and measurement variables".8 External validity is a matter of judgment, which depends on the characteristics of the patients included in the trial, the setting, the treatment regimens, and the outcomes assessed (box).8 In recent years large meta-analyses based on data from individual patients have shown that important differences in treatment effects may exist between patient groups and settings. For example, antihypertensive treatment reduces total mortality in middle aged patients with hypertension, but this may not be the case in elderly people.25 The benefits of fibrinolytic treatment in suspected acute myocardial infarction has been shown to decrease linearly with the delay between the start of symptoms and the initiation of treatment.26 In trials of cholesterol lowering drugs the benefits of a reduction in non-fatal myocardial infarction and mortality due to coronary heart disease depends on the reduction in total cholesterol concentration and the duration of follow up.27 At the very least, therefore, assessment of a trial's applicability requires adequate information about the characteristics of the participants.


    Quality of reporting

The assessment of the methodological quality of a trial is intertwined with the quality of reporting---that is, the extent to which a report provides information about the design, conduct, and analysis of the trial.4 Reports often omit important methodological details. For example, only 1 of 122 randomised trials of selective serotonin reuptake inhibitors specified the method of randomisation.28 A widely used approach to this problem is to assume that the quality was inadequate unless the information to the contrary is provided (the "guilty until proved innocent" approach). This is often justified because faulty reporting generally reflects faulty methods. 20 29 A well conducted but badly reported trial will, however, be misclassified. An alternative approach is to explicitly assess the quality of the reporting rather than the adequacy of the methods. This is also problematic because a biased but well reported trial will receive full credit.30 The adoption of guidelines on the reporting of clinical trials has recently improved this situation for several journals, 31 32 but deficiencies in reporting will continue to be confused with deficiencies in design, conduct, and analysis.


    Assessing trial quality

How the quality of trials should be assessed is being debated. Quality scales combine information on several features in a single numerical value, whereas the component approach examines key dimensions individually, without calculation of a score. Moher et al reviewed the use of quality scores in systematic reviews published in medical journals and the Cochrane database of systematic reviews.33 Trial quality was assessed in 78 (38%) of the 204 reviews from journals, of which 20 (26%) used components and 52 (67%) used scales. By contrast, all 36 reviews from the database assessed quality, of which 33 (92%) used components and none used scales.

Scales vary considerably in dimensions covered and complexity.4 Many scales include items for which there is little evidence that they are related to the internal validity of a trial. For example, a widely used instrument includes items related to the presentation of data and the organisation of the trial.34 Unsurprisingly, different scales can lead to discordant results. This was shown in a study in which 25 different scales were used to assess 17 trials comparing low molecular weight heparin with standard heparin for thromboprophylaxis.5 With some scales, the relative risks of the "high quality" trials were close to unity and not statistically significant, indicating that low molecular weight heparin was not superior to standard heparin, whereas the "low quality" trials assessed by these scales showed better protection with the low molecular weight heparin. With other scales the opposite was the case: high quality trials indicated that low molecular weight heparin was superior to standard heparin, whereas low quality trials found no significant difference.5

When the association of effect estimates with quality scores is examined, interpretation of results is difficult. In the absence of an association there are three possible explanations35: there is no association with any of the components; there are associations with one or several components, but these components have so little weight that the effects are lost in the summary score; or there are associations with two or more components, but these cancel out so that no association is found with the overall score. On the other hand, if treatment effects do vary with quality scores then investigators will have to identify the component or components that are responsible for this association to interpret this finding.

The analysis of individual components of trial quality overcomes many of the shortcomings of composite scores. The component approach takes into account that the importance of individual quality domains, and the direction of potential biases associated with these domains, varies between the contexts in which trials are performed.
 
(Credit: MARK OLDROYD)



    Incorporating study quality into meta-analysis

It makes intuitive sense to take into account information on the quality of studies when doing systematic reviews. One approach is to exclude trials that fail to meet some standard of quality. This may often be justified but could exclude studies that might contribute valid information. It may therefore be prudent to exclude only trials with gross deficiencies in design---for example, those that clearly failed to study comparable groups. The possible influence of study quality on effect estimates should, however, always be examined in a given set of included studies. Several approaches have been proposed for this purpose.

Quality as a weight in statistical pooling
The most radical approach is to directly incorporate information on study quality as weighting factors in the analysis. Study weights can be multiplied by quality scores, thus increasing the weight of trials deemed to be of high quality and decreasing the weight of those of low quality. 3 21 A trial with a quality score of 40 out of 100 will thus get the same weight in the analysis as a trial with half the amount of information but a quality score of 80.

Weighting by quality scores is problematic for several reasons. As mentioned, the choice of the scale influences the weight of individual studies in the analysis, and the combined effect estimate and its confidence interval therefore depend on the scale. However, there is no reason why study quality should modify the precision of estimates. Poor studies are still included. Thus any bias associated with poor methodology is only reduced, not removed. Including both good and poor studies may also increase heterogeneity of estimated effects across trials and may reduce the credibility of a systematic review. The incorporation of quality scores as weights lacks statistical or empirical justification.3

Sensitivity analysis
The robustness of the findings of a meta-analysis to different assumptions should always be examined in a thorough sensitivity analysis. An assessment of the influence of methodological quality should be part of this process. Simple stratified analyses and meta-regression models are useful for exploring associations between treatment effects and study characteristics. Quality summary scores or categorical data on individual components can be used for this purpose. For the reasons discussed the authors recommend that sensitivity analysis should be based on the components of study quality that are considered important in the context of a given meta-analysis. Other approaches, such as plotting effect estimates against quality scores or performing cumulative meta-analysis in order of quality, are also affected by the problems surrounding composite scales. 3 36

Conclusions
There is ample evidence that many trials are methodologically weak and increasing evidence that deficiencies translate into biased findings of systematic reviews. The assessment of the methodological quality of controlled trials and the conduct of sensitivity analyses should therefore be considered routine procedures in systematic reviews and meta-analysis. Although composite quality scales may provide a useful overall assessment when comparing populations of trials, such scales should generally not be used to identify trials of apparent low quality or high quality in a given systematic review. Rather, the relevant methodological aspects should be identified a priori and assessed individually. This should include the generation and concealment of treatment allocation, blinding, and handling of attrition in the analysis. Other ways of investigating and dealing with bias in systematic reviews will be discussed and illustrated later in this series.37



View larger version (71K):
[in this window]
[in a new window]
  		Systematic Reviews in Health Care: Meta-analysis in Context can be purchased through the BMJ Bookshop (www.bmjbookshop.com); further information and updates for the book are available on www.systematicreviews.uom


    Acknowledgments

We thank Ken Schulz and Lise Kjaergard for unpublished data and Iain Chalmers for useful comments on an earlier version of this paper.

    Footnotes

Series editor: Matthias Egger

Funding: PJ is supported by the Swiss National Science Foundation. The work on trial quality in Bristol was supported by the NHS Research and Development Programme.

Competing interests: None declared.


    References

1. Clarke M, Oxman AD, eds. Cochrane reviewers' handbook 4.0. In: Cochrane Collaboration. Cochrane Library. Oxford: Update Software, 1999.
2. Cook DJ, Sackett DL, Spitzer WO. Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam consultation on meta-analysis. J Clin Epidemiol 1995; 48: 167-171[CrossRef][Medline].
3. Detsky AS, Naylor CD, O'Rourke K, McGeer AJ, L'Abbé KA. Incorporating variations in the quality of individual randomized trials into meta-analysis. J Clin Epidemiol 1992; 45: 255-265[CrossRef][Medline].
4. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Controlled Clin Trials 1995; 16: 62-73[Medline].
5. Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trial for meta-analysis. JAMA 1999; 282: 1054-1060[Abstract/Free Full Text].
6. Jüni P, Altman DG, Egger M. Assessing the quality of controlled clinical trials. In: Egger M, Davey Smith G, Altman DG, eds. Systematic reviews in health care: meta-analysis in context 2nd ed. London: BMJ Books, 2001.
7. Campbell DT. Factors relevant to the validity of experiments in social settings. Psychol Bull 1957; 54: 297-312[CrossRef][Medline].
8. Campbell DT, Stanley JC. Experimental and quasi-experimental designs for research on teaching. In: Gage NL, ed. Handbook of research on teaching. Chicago: Rand McNally, 1963:171-246.
9. Murphy EA. The logic of medicine. Baltimore: Johns Hopkins University Press, 1976.
10. Altman DG, Bland JM. Treatment allocation in controlled trials: why randomise? BMJ 1999; 318: 1209[Free Full Text].
11. Altman DG. Randomisation. Essential for reducing bias. BMJ 1991; 302: 1481-1482.
12. Keirse MJ. Amniotomy or oxytocin for induction of labor. Re-analysis of a randomized controlled trial. Acta Obstet Gynecol Scand 1988; 67: 731-735[Medline].
13. Schulz KF. Randomised trials, human nature, and reporting guidelines. Lancet 1996; 348: 596-598[CrossRef][Medline].
14. Schulz KF. Subverting randomization in controlled trials. JAMA 1995; 274: 1456-1458[Abstract].
15. Noseworthy JH, Ebers GC, Vandervoort MK, Farquhar RE, Yetisir E, Roberts R. The impact of blinding on the results of a randomized, placebo-controlled multiple sclerosis clinical trial. Neurology 1994; 44: 16-20[Abstract/Free Full Text].
16. Sackett DL, Gent M. Controversy in counting and attributing events in clinical trials. N Engl J Med 1979; 301: 1410-1412[Medline].
17. Coronary Drug Project Research Group. Influence of adherence to treatment and response of cholesterol on mortality in the CDP. N Engl J Med 1980; 303: 1038-1041[Abstract].
18. May GS, Demets DL, Friedman LM, Furberg C, Passamani E. The randomized clinical trial: bias in analysis. Circulation 1981; 64: 669-673[Abstract/Free Full Text].
19. Hollis S, Campbell F. What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ 1999; 319: 670-674[Abstract/Free Full Text].
20. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408-412[Abstract].
21. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998; 352: 609-613[CrossRef][Medline].
22. Jüni P, Tallon D, Egger M. `Garbage in - garbage out'? Assessment of the quality of controlled trials in meta-analyses published in leading journals. In: Proceedings of the 3rd symposium on systematic reviews: beyond the basics, St Catherine's College, Oxford. Oxford: Centre for Statistics in Medicine, 2000:19.
23. Kjaergard LL, Villumsen J, Gluud C. Quality of randomised clinical trials affects estimates of intervention efficacy. In: Proceedings of the 7th Cochrane colloquium. Universita S.Tommaso D'Aquino, Rome. Milan: Centro Cochrane Italiano, 1999:57 (poster B10).
24. Devereaux PJ, Manns BJ, Ghali WA, Quan H, Lacchetti C, Mouton VM, et al. Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. JAMA 2001; 285: 2000-2003[Abstract/Free Full Text].
25. Gueyffier F, Bulpitt C, Boissel JP, Schron E, Ekbom T, Fagard R, et al. Antihypertensive drugs in very old people: a subgroup meta-analysis of randomised controlled trials. Lancet 1999; 353: 796.
26. Fibrinolytic Therapy Trialists' (FTT) Collaborative Group. Indications for fibrinolytic therapy in suspected acute myocardial infarction: collaborative overview of early mortality and major morbidity results from all randomised trials of more than 1000 patients. Lancet 1994; 343: 311-322[CrossRef][Medline].
27. Thompson SG. Controversies in meta-analysis: the case of the trials of serum cholesterol reduction. Stat Methods Med Res 1993; 2: 173-192[Medline].
28. Hotopf M, Lewis G, Normand C. Putting trials on trial---the costs and consequences of small trials in depression: a systematic review of methodology. J Epidemiol Community Health 1997; 51: 354-358[Abstract].
29. Liberati A, Himel HN, Chalmers TC. A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol 1986; 4: 942-951[Abstract/Free Full Text].
30. Feinstein AR. Meta-analysis: statistical alchemy for the 21st century. J Clin Epidemiol 1995; 48: 71-79[CrossRef][Medline].
31. Moher D, Jones A, Lepage L. Use of the CONSORT statement and quality of reports of randomized trials. JAMA 2001; 285: 1987-1991[Abstract/Free Full Text].
32. Egger M, Jüni P, Bartlett C. Value of flow diagrams in reports of randomized controlled trials. JAMA 2001; 285: 1996-1999[Abstract/Free Full Text].
33. Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A, et al. Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technol Assess 1999;i3(12).
34. Chalmers TC, Smith H, Blackburn B, Silverman B, Schroeder B, Reitman D, et al. A method for assessing the quality of a randomized control trial. Controlled Clin Trials 1981; 2: 31-49[CrossRef][Medline].
35. Greenland S. Quality scores are useless and potentially misleading. Am J Epidemiol 1994; 140: 300-302[Free Full Text].
36. Linde K, Scholz M, Ramirez G, Clausius N, Melchart D, Jonas WB. Impact of study quality on outcome in placebo-controlled trials of homeopathy. J Clin Epidemiol 1999; 52: 631-636[CrossRef][Medline].
37. Sterne JAC, Egger M, Davey Smith G. Investigating and dealing with publication and other biases in meta-analysis. BMJ 2001 (in press).

© BMJ 2001
Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati    What's this?

Related external webpages:

NHS Health Technology Assessment Programme

This article has been cited by other articles:

  • Gonzalez, R., Zamora, J., Gomez-Camarero, J., Molinero, L.-M., Banares, R., Albillos, A. (2008). Meta-analysis: Combination Endoscopic and Drug Therapy to Prevent Variceal Rebleeding in Cirrhosis. ANN INTERN MED 149: 109-122 [Abstract] [Full text]  
  • Lourenco, T., Pickard, R., Vale, L., Grant, A., Fraser, C., MacLennan, G., N'Dow, J., and the Benign Prostatic Enlargement team, (2008). Alternative approaches to endoscopic ablation for benign enlargement of the prostate: systematic review of randomised controlled trials. BMJ 337: a449-a449 [Abstract] [Full text]  
  • Chan, E. (2008). Quality of Efficacy Research in Complementary and Alternative Medicine. JAMA 299: 2685-2686 [Full text]  
  • Higgins, J. P., White, I. R, Wood, A. M (2008). Imputation methods for missing outcome data in meta-analysis of clinical trials. Clin Trials 5: 225-239 [Abstract]  
  • Salanti, G., Higgins, J. P., Ades, A., Ioannidis, J. P. (2008). Evaluation of networks of randomized trials. Stat Methods Med Res 17: 279-301 [Abstract]  
  • Karanicolas, P. J., Bhandari, M., Taromi, B., Akl, E. A., Bassler, D., Alonso-Coello, P., Rigau, D., Bryant, D., Smith, S. E., Walter, S. D., Guyatt, G. H. (2008). Blinding of Outcomes in Trials of Orthopaedic Trauma: An Opportunity to Enhance the Validity of Clinical Trials. JBJS 90: 1026-1033 [Abstract] [Full text]  
  • Wykes, T., Steel, C., Everitt, B., Tarrier, N. (2008). Cognitive Behavior Therapy for Schizophrenia: Effect Sizes, Clinical Models, and Methodological Rigor. Schizophr Bull 34: 523-537 [Abstract] [Full text]  
  • Abrahamyan, L., Johnson, S. R., Beyene, J., Shah, P. S., Feldman, B. M. (2008). Quality of randomized clinical trials in juvenile idiopathic arthritis. Rheumatology (Oxford) 47: 640-645 [Abstract] [Full text]  
  • Patsopoulos, N. A, Evangelou, E., Ioannidis, J. P. (2008). Sensitivity of between-study heterogeneity in meta-analysis: proposed metrics and empirical evaluation. Int J Epidemiol 0: dyn065v1-dyn065 [Abstract] [Full text]  
  • Djulbegovic, B., Kumar, A., Soares, H. P., Hozo, I., Bepler, G., Clarke, M., Bennett, C. L. (2008). Treatment Success in Cancer: New Cancer Treatment Successes Identified in Phase 3 Randomized Controlled Trials Conducted by the National Cancer Institute-Sponsored Cooperative Oncology Groups, 1955 to 2006. Arch Intern Med 168: 632-642 [Abstract] [Full text]  
  • Manheimer, E., Zhang, G., Udoff, L., Haramati, A., Langenberg, P., Berman, B. M, Bouter, L. M (2008). Effects of acupuncture on rates of pregnancy and live birth among women undergoing in vitro fertilisation: systematic review and meta-analysis. BMJ 336: 545-549 [Abstract] [Full text]  
  • Wieland, D., Ferrucci, L. (2008). Multidimensional Geriatric Assessment: Back to the Future. J. Gerontol. A Biol. Sci. Med. Sci. 63: 272-274 [Full text]  
  • Huss, A., Stuck, A. E., Rubenstein, L. Z., Egger, M., Clough-Gorr, K. M. (2008). Multidimensional Geriatric Assessment: Back to the Future Multidimensional Preventive Home Visit Programs for Community-Dwelling Older Adults: A Systematic Review and Meta-Analysis of Randomized Controlled Trials. J. Gerontol. A Biol. Sci. Med. Sci. 63: 298-307 [Abstract] [Full text]  
  • Crossley, N. A., Sena, E., Goehler, J., Horn, J., van der Worp, B., Bath, P. M.W., Macleod, M., Dirnagl, U. (2008). Empirical Evidence of Bias in the Design of Experimental Stroke Studies: A Metaepidemiologic Approach. Stroke 39: 929-934 [Abstract] [Full text]  
  • Boutron, I., Moher, D., Altman, D. G., Schulz, K. F., Ravaud, P., for the CONSORT Group, (2008). Extending the CONSORT Statement to Randomized Trials of Nonpharmacologic Treatment: Explanation and Elaboration. ANN INTERN MED 148: 295-309 [Abstract] [Full text]  
  • Hollin, C. R. (2008). Evaluating offending behaviour programmes: Does only randomization glister?. Criminal Justice 8: 89-106 [Abstract]  
  • Coomarasamy, A., Afnan, M., Cheema, D., van der Veen, F., Bossuyt, P. M.M., van Wely, M. (2008). Urinary hMG versus recombinant FSH for controlled ovarian hyperstimulation following an agonist long down-regulation protocol in IVF or ICSI treatment: a systematic review and meta-analysis. Hum Reprod 23: 310-315 [Abstract] [Full text]  
  • Whelan, A. M., Jurgens, T. M., Lord, L. (2008). Evaluating the Quality of Randomized Controlled Trials that Examine the Efficacy of Natural Health Products: A Systematic Review of Critical Appraisal Instruments. Evid Based Complement Alternat Med 0: nem186v1-nem186 [Abstract] [Full text]  
  • Tarrier, N., Taylor, K., Gooding, P. (2008). Cognitive-Behavioral Interventions to Reduce Suicide Behavior: A Systematic Review and Meta-Analysis. Behav Modif 32: 77-108 [Abstract]  
  • Garg, A. X., Greene, T., Levin, N. W. (2008). A well-conducted randomized trial that establishes no benefit of therapy is an important medical advance. Nephrol Dial Transplant 23: 52-55 [Full text]  
  • Garg, A. X., Hackam, D., Tonelli, M. (2008). Systematic Review and Meta-analysis: When One Study Is Just not Enough. CJASN 3: 253-260 [Full text]  
  • Trelle, S., Reichenbach, S., Juni, P. (2007). Chondroitin for Osteoarthritis of the Knee or Hip. ANN INTERN MED 147: 884-885 [Full text]  
  • Kastrati, A., Dibra, A., Spaulding, C., Laarman, G. J., Menichelli, M., Valgimigli, M., Di Lorenzo, E., Kaiser, C., Tierala, I., Mehilli, J., Seyfarth, M., Varenne, O., Dirksen, M. T., Percoco, G., Varricchio, A., Pittl, U., Syvanne, M., Suttorp, M. J., Violini, R., Schomig, A. (2007). Meta-analysis of randomized trials on drug-eluting stents vs. bare-metal stents in patients with acute myocardial infarction. Eur Heart J 28: 2706-2713 [Abstract] [Full text]  
  • Raveendran, N. S, Tharyan, P., Alexander, J., Adams, C. E., TREC-India II Collaborative Group, (2007). Rapid tranquillisation in psychiatric emergency settings in India: pragmatic randomised controlled trial of intramuscular olanzapine versus intramuscular haloperidol plus promethazine. BMJ 335: 865-865 [Abstract] [Full text]  
  • Reddy, M., Gill, S. S., Kalkar, S. R., Wu, W., Anderson, P. J., Rochon, P. A. (2007). Oral Drug Therapy for Multiple Neglected Tropical Diseases: A Systematic Review. JAMA 298: 1911-1924 [Abstract] [Full text]  
  • Elm, E. v., Altman, D. G, Egger, M., Pocock, S. J, Gotzsche, P. C, Vandenbroucke, J. P, STROBE Initiative, (2007). Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ 335: 806-808 [Full text]  
  • Lowe, C. J M., Barker, K. L, Dewey, M., Sackley, C. M (2007). Effectiveness of physiotherapy exercise after knee arthroplasty for osteoarthritis: systematic review and meta-analysis of randomised controlled trials. BMJ 335: 812-812 [Abstract] [Full text]  
  • von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gotzsche, P. C., Vandenbroucke, J. P., for the STROBE Initiative, (2007). The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies. ANN INTERN MED 147: 573-577 [Abstract] [Full text]  
  • Schomig, A., Dibra, A., Windecker, S., Mehilli, J., Suarez de Lezo, J., Kaiser, C., Park, S.-J., Goy, J.-J., Lee, J.-H., Di Lorenzo, E., Wu, J., Juni, P., Pfisterer, M. E., Meier, B., Kastrati, A. (2007). A Meta-Analysis of 16 Randomized Trials of Sirolimus-Eluting Stents Versus Paclitaxel-Eluting Stents in Patients With Coronary Artery Disease. J Am Coll Cardiol 50: 1373-1380 [Abstract] [Full text]  
  • Shang, A., Huwiler, K., Nartey, L., Juni, P., Egger, M. (2007). Placebo-controlled trials of Chinese herbal medicine and conventional medicine comparative study. Int J Epidemiol 36: 1086-1092 [Abstract] [Full text]  
  • Togni, M., Eber, S., Widmer, J., Billinger, M., Wenaweser, P., Cook, S., Vogel, R., Seiler, C., Eberli, F. R., Maier, W., Corti, R., Roffi, M., Luscher, T. F., Garachemani, A., Hess, O. M., Wandel, S., Meier, B., Juni, P., Windecker, S. (2007). Impact of Vessel Size on Outcome After Implantation of Sirolimus-Eluting and Paclitaxel-Eluting Stents: A Subgroup Analysis of the SIRTAX Trial. J Am Coll Cardiol 50: 1123-1131 [Abstract] [Full text]  
  • Martin, J. L. R., Sainz-Pardo, M., Furukawa, T. A., Martin-Sanchez, E., Seoane, T., Galan, C. (2007). Review: Benzodiazepines in generalized anxiety disorder: heterogeneity of outcomes based on a systematic review and meta-analysis of clinical trials. J Psychopharmacol 21: 774-782 [Abstract]  
  • Aaltonen, S., Karjalainen, H., Heinonen, A., Parkkari, J., Kujala, U. M. (2007). Prevention of Sports Injuries: Systematic Review of Randomized Controlled Trials. Arch Intern Med 167: 1585-1592 [Abstract] [Full text]  
  • Pildal, J, Hrobjartsson, A, Jorgensen, K., Hilden, J, Altman, D., Gotzsche, P. (2007). Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 36: 847-857 [Abstract] [Full text]  
  • Isakow, W., Morrow, L. E., Kollef, M. H. (2007). Probiotics for Preventing and Treating Nosocomial Infections: Review of Current Evidence and Recommendations. Chest 132: 286-294 [Abstract] [Full text]  
  • Boutron, I., Ravaud, P., Nizard, R. (2007). The design and assessment of prospective randomised, controlled trials in orthopaedic surgery. J Bone Joint Surg Br 89-B: 858-863 [Abstract] [Full text]  
  • O. ter Kuile, F., van Eijk, A. M., Filler, S. J. (2007). Effect of Sulfadoxine-Pyrimethamine Resistance on the Efficacy of Intermittent Preventive Therapy for Malaria Control During Pregnancy: A Systematic Review. JAMA 297: 2603-2616 [Abstract] [Full text]  
  • Manheimer, E., Linde, K., Lao, L., Bouter, L. M., Berman, B. M. (2007). Meta-analysis: Acupuncture for Osteoarthritis of the Knee. ANN INTERN MED 146: 868-877 [Abstract] [Full text]  
  • Karanicolas, P. J., Davies, E., Kunz, R., Briel, M., Koka, H. P., Payne, D. M., Smith, S. E., Hsu, H.-P., Lin, P.-W., Bloechle, C., Paquet, K.-J., Guyatt, G. H. (2007). The Pylorus: Take It or Leave It? Systematic Review and Meta-Analysis of Pylorus-Preserving versus Standard Whipple Pancreaticoduodenectomy for Pancreatic or Periampullary Cancer. Ann. Surg. Oncol. 14: 1825-1834 [Abstract] [Full text]  
  • Hrobjartsson, A, Forfang, E, Haahr, M., Als-Nielsen, B, Brorson, S (2007). Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding. Int J Epidemiol 36: 654-663 [Abstract] [Full text]  
  • Sanderson, S., Tatt, I. D, Higgins, J. P. (2007). Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 36: 666-676 [Abstract] [Full text]  
  • Abdel-Latif, A., Bolli, R., Tleyjeh, I. M., Montori, V. M., Perin, E. C., Hornung, C. A., Zuba-Surma, E. K., Al-Mallah, M., Dawn, B. (2007). Adult Bone Marrow-Derived Cells for Cardiac Repair: A Systematic Review and Meta-analysis. Arch Intern Med 167: 989-997 [Abstract] [Full text]  
  • Freidlin, B., Korn, E. L., Hunsberger, S., Gray, R., Saxman, S., Zujewski, J. A. (2007). Proposal for the Use of Progression-Free Survival in Unblinded Randomized Trials. JCO 25: 2122-2126 [Abstract] [Full text]  
  • Schulz, H., Bohlius, J. F., Trelle, S., Skoetz, N., Reiser, M., Kober, T., Schwarzer, G., Herold, M., Dreyling, M., Hallek, M., Engert, A. (2007). Immunochemotherapy With Rituximab and Overall Survival in Patients With Indolent or Mantle Cell Lymphoma: A Systematic Review and Meta-analysis. JNCI J Natl Cancer Inst 99: 706-714 [Abstract] [Full text]  
  • Eisenberg, J. N.S., Scott, J. C., Porco, T. (2007). Integrating Disease Control Strategies: Balancing Water Sanitation and Hygiene Interventions to Reduce Diarrheal Disease Burden. Am. J. Public Health 97: 846-852 [Abstract] [Full text]  
  • Reichenbach, S., Sterchi, R., Scherer, M., Trelle, S., Burgi, E., Burgi, U., Dieppe, P. A., Juni, P. (2007). Meta-analysis: Chondroitin for Osteoarthritis of the Knee or Hip. ANN INTERN MED 146: 580-590 [Abstract] [Full text]  
  • Clasen, T., Schmidt, W.-P., Rabie, T., Roberts, I., Cairncross, S. (2007). Interventions to improve water quality for preventing diarrhoea: systematic review and meta-analysis. BMJ 334: 782-782 [Abstract] [Full text]  
  • Kastrati, A., Mehilli, J., Pache, J., Kaiser, C., Valgimigli, M., Kelbaek, H., Menichelli, M., Sabate, M., Suttorp, M. J., Baumgart, D., Seyfarth, M., Pfisterer, M. E., Schomig, A. (2007). Analysis of 14 Trials Comparing Sirolimus-Eluting Stents with Bare-Metal Stents. NEJM 356: 1030-1039 [Abstract] [Full text]  
  • Poolman, R. W., Struijs, P. A.A., Krips, R., Sierevelt, I. N., Marti, R. K., Farrokhyar, F., Bhandari, M. (2007). Reporting of Outcomes in Orthopaedic Randomized Trials: Does Blinding of Outcome Assessors Matter?. JBJS 89: 550-558 [Abstract] [Full text]  
  • Burris, N., Schwartz, K., Tang, C.-M., Jafri, M. S., Schmitt, J., Kwon, M. H., Toshinaga, O., Gu, J., Brown, J., Brown, E., Pierson, R. III, Poston, R. (2007). Catheter-based infrared light scanner as a tool to assess conduit quality in coronary artery bypass surgery. J. Thorac. Cardiovasc. Surg. 133: 419-427 [Abstract] [Full text]  
  • Zintzaras, E., Kaditis, A. G. (2007). Sleep-Disordered Breathing and Blood Pressure in Children: A Meta-analysis. Arch Pediatr Adolesc Med 161: 172-178 [Abstract] [Full text]  
  • Flores-Mir, C., Major, P. W. (2006). A systematic review of cephalometric facial soft tissue changes with the Activator and Bionator appliances in Class II division 1 subjects. Eur J Orthod 28: 586-593 [Abstract] [Full text]  
  • Chang, D. C. (2006). Meta-analysis in Surgery--Invited Critique. Arch Surg 141: 1131-1131 [Full text]  
  • Reeves, B. C (2006). Parachute approach to evidence based medicine: as obvious as ABC.. BMJ 333: 807-807 [Full text]  
  • Winzenberg, T., Shaw, K., Fryer, J., Jones, G. (2006). Effects of calcium supplementation on bone density in healthy children: meta-analysis of randomised controlled trials. BMJ 333: 775- [Abstract] [Full text]  
  • Jorgensen, A. W, Hilden, J., Gotzsche, P. C (2006). Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ 333: 782- [Abstract] [Full text]  
  • Haahr, M. T., Hrobjartsson, A. (2006). Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors. Clin Trials 3: 360-365 [Abstract]  
  • Tiruvoipati, R., Balasubramanian, S. P., Atturu, G., Peek, G. J., Elbourne, D. (2006). Improving the quality of reporting randomized controlled trials in cardiothoracic surgery: The way forward.. J. Thorac. Cardiovasc. Surg. 132: 233-240 [Abstract] [Full text]  
  • Reeves, B. C. (2006). Limitations of analyses of effectiveness using observational data. Eur Heart J 27: 1642-1643 [Full text]  
  • Costello, M. F., Chapman, M., Conway, U. (2006). A systematic review and meta-analysis of randomized controlled trials on metformin co-administration during gonadotrophin ovulation induction or IVF in women with polycystic ovary syndrome. Hum Reprod 21: 1387-1399 [Abstract] [Full text]  
  • Ioannidis, J. P A (2006). Commentary: Grading the credibility of molecular evidence for complex diseases. Int J Epidemiol 35: 572-578 [Full text]  
  • Kubo, A., Corley, D. A. (2006). Body Mass Index and Adenocarcinomas of the Esophagus or Gastric Cardia: A Systematic Review and Meta-analysis.. Cancer Epidemiol. Biomarkers Prev. 15: 872-878 [Abstract] [Full text]  
  • Reid, C. M., Martin, R. M., Sterne, J. A. C., Davies, A. N., Hanks, G. W. (2006). Oxycodone for Cancer-Related Pain: Meta-analysis of Randomized Controlled Trials.. Arch Intern Med 166: 837-843 [Abstract] [Full text]  
  • Al-Mallah, M. H., Tleyjeh, I. M., Abdel-Latif, A. A., Weaver, W. D. (2006). Angiotensin-Converting Enzyme Inhibitors in Coronary Artery Disease and Preserved Left Ventricular Systolic Function: A Systematic Review and Meta-Analysis of Randomized Controlled Trials. J Am Coll Cardiol 47: 1576-1583 [Abstract] [Full text]  
  • Danchin, N., Cucherat, M., Thuillez, C., Durand, E., Kadri, Z., Steg, P. G. (2006). Angiotensin-Converting Enzyme Inhibitors in Patients With Coronary Artery Disease and Absence of Heart Failure or Left Ventricular Systolic Dysfunction: An Overview of Long-term Randomized Controlled Trials.. Arch Intern Med 166: 787-796 [Abstract] [Full text]  
  • Kanna, B., Mishra, N. (2006). Efficacy and Safety of Inhaled Insulin Therapy. ANN INTERN MED 144: 533-533 [Full text]  
  • Latthe, P., Mignini, L., Gray, R., Hills, R., Khan, K. (2006). Factors predisposing women to chronic pelvic pain: systematic review. BMJ 332: 749-755 [Abstract] [Full text]  
  • Gluud, L. L. (2006). Bias in Clinical Intervention Research. Am J Epidemiol 163: 493-501 [Abstract] [Full text]  
  • Piaggio, G., Elbourne, D. R., Altman, D. G., Pocock, S. J., Evans, S. J. W., for the CONSORT Group, (2006). Reporting of Noninferiority and Equivalence Randomized Trials: An Extension of the CONSORT Statement. JAMA 295: 1152-1160 [Abstract] [Full text]  
  • Rutjes, A. W.S., Reitsma, J. B., Di Nisio, M., Smidt, N., van Rijn, J. C., Bossuyt, P. M.M. (2006). Evidence of bias and variation in diagnostic accuracy studies.. CMAJ 174: 469-476 [Abstract] [Full text]  
  • Loscalzo, J. (2005). Clinical Trials in Cardiovascular Medicine in an Era of Marginal Benefit, Bias, and Hyperbole. Circulation 112: 3026-3029 [Full text]  
  • Braslow, J. T., Duan, N., Starks, S. L., Polo, A., Bromley, E., Wells, K. B. (2005). Generalizability of Studies on Mental Health Treatment and Outcomes, 1981 to 1996. Psychiatr. Serv. 56: 1261-1268 [Abstract] [Full text]  
  • Oliver, S., Harden, A., Rees, R., Shepherd, J., Brunton, G., Garcia, J., Oakley, A. (2005). An Emerging Framework for Including Different Types of Evidence in Systematic Reviews for Public Policy. Evaluation 11: 428-446 [Abstract]  
  • Windecker, S., Remondino, A., Eberli, F. R., Juni, P., Raber, L., Wenaweser, P., Togni, M., Billinger, M., Tuller, D., Seiler, C., Roffi, M., Corti, R., Sutsch, G., Maier, W., Luscher, T., Hess, O. M., Egger, M., Meier, B. (2005). Sirolimus-Eluting and Paclitaxel-Eluting Stents for Coronary Revascularization. NEJM 353: 653-662 [Abstract] [Full text]  
  • Kastrati, A., Dibra, A., Eberle, S., Mehilli, J., Suarez de Lezo, J., Goy, J.-J., Ulm, K., Schomig, A. (2005). Sirolimus-Eluting Stents vs Paclitaxel-Eluting Stents in Patients With Coronary Artery Disease: Meta-analysis of Randomized Trials. JAMA 294: 819-825 [Abstract] [Full text]  
  • Dawes, M. (2005). Combing and combating head lice. BMJ 331: 362-363 [Full text]  
  • van der Meer, V., Neven, A. K., van den Broek, P. J, Assendelft, W. J J (2005). Diagnostic value of C reactive protein in infections of the lower respiratory tract: systematic review. BMJ 331: 26- [Abstract] [Full text]  
  • Nestaas, E, Bangstad, H-J, Sandvik, L, Wathne, K-O (2005). Aminoglycoside extended interval dosing in neonates is safe and effective: a meta-analysis. Arch. Dis. Child. Fetal Neonatal Ed. 90: F294-f300 [Abstract] [Full text]  
  • Hewitt, C., Hahn, S., Torgerson, D. J, Watson, J., Bland, J M. (2005). Adequacy and reporting of allocation concealment: review of recent trials published in four general medical journals. BMJ 330: 1057-1058 [Full text]  

Rapid Responses:

Read all Rapid Responses

Bias against quality scales
Terry P Klassen
bmj.com, 9 Jul 2001 [Full text]
Authors’ response: Why scales are unhelpful for assessing quality of trials
Peter Jüni
bmj.com, 24 Aug 2001 [Full text]
Effect of intra and inter-observer variation on systematic reviews of controlled clinical trials
J Bernades, et al.
bmj.com, 5 Sep 2001 [Full text]

Student BMJ

Intimate examinations

Israeli students are refusing to perform intimate examinations on anaesthetised women without their informed consent.

www.student.bmj.com

Listen to the latest BMJ Interview