Intended for healthcare professionals

CCBYNC Open access

Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study

BMJ 2009; 339 doi: (Published 13 August 2009) Cite this as: BMJ 2009;339:b3128
  1. Britta Tendal, PhD student1,
  2. Julian P T Higgins, senior statistician4,
  3. Peter Jüni, head of division23,
  4. Asbjørn Hróbjartsson, senior researcher1,
  5. Sven Trelle, associate director23,
  6. Eveline Nüesch, PhD student23,
  7. Simon Wandel, PhD student23,
  8. Anders W Jørgensen, PhD student1,
  9. Katarina Gesser, PhD student5,
  10. Søren Ilsøe-Kristensen, PhD student5,
  11. Peter C Gøtzsche, director1
  1. 1Nordic Cochrane Centre, Rigshospitalet, Dept 3343, Blegdamsvej 9, DK-2100 Copenhagen, Denmark
  2. 2Institute of Social and Preventive Medicine, University of Bern, Switzerland
  3. 3CTU Bern, Bern University Hospital, Switzerland
  4. 4MRC Biostatistics Unit, Institute of Public Health, University of Cambridge, Cambridge
  5. 5Faculty of Pharmaceutical Sciences, University of Copenhagen, Denmark
  1. Correspondence to: B Tendal bt{at}
  • Accepted 11 March 2009


Objective To study the inter-observer variation related to extraction of continuous and numerical rating scale data from trial reports for use in meta-analyses.

Design Observer agreement study.

Data sources A random sample of 10 Cochrane reviews that presented a result as a standardised mean difference (SMD), the protocols for the reviews and the trial reports (n=45) were retrieved.

Data extraction Five experienced methodologists and five PhD students independently extracted data from the trial reports for calculation of the first SMD result in each review. The observers did not have access to the reviews but to the protocols, where the relevant outcome was highlighted. The agreement was analysed at both trial and meta-analysis level, pairing the observers in all possible ways (45 pairs, yielding 2025 pairs of trials and 450 pairs of meta-analyses). Agreement was defined as SMDs that differed less than 0.1 in their point estimates or confidence intervals.

Results The agreement was 53% at trial level and 31% at meta-analysis level. Including all pairs, the median disagreement was SMD=0.22 (interquartile range 0.07-0.61). The experts agreed somewhat more than the PhD students at trial level (61% v 46%), but not at meta-analysis level. Important reasons for disagreement were differences in selection of time points, scales, control groups, and type of calculations; whether to include a trial in the meta-analysis; and data extraction errors made by the observers. In 14 out of the 100 SMDs calculated at the meta-analysis level, individual observers reached different conclusions than the originally published review.

Conclusions Disagreements were common and often larger than the effect of commonly used treatments. Meta-analyses using SMDs are prone to observer variation and should be interpreted with caution. The reliability of meta-analyses might be improved by having more detailed review protocols, more than one observer, and statistical expertise.


  • Contributors: All authors had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: BT, PCG, JPTH, PJ. Acquisition of data: all authors. Analysis and interpretation of data: BT, PCG, JPTH, PJ, EN. Drafting of the manuscript: BT. Critical revision of the manuscript for important intellectual content: all authors. Statistical analysis: BT, PCG, JPTH, EN. Administrative, technical, or material support: BT, PCG. Study guarantor: PCG.

  • Funding: This study is part of a PhD funded by IMK Charitable Fund and the Nordic Cochrane Centre. The sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The researchers were independent from the funders.

  • Competing interests: None declared.

  • Ethical approval: Not required

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: and

View Full Text