Intended for healthcare professionals

Endgames Statistical Question

Meta-analyses V

BMJ 2011; 342 doi: https://doi.org/10.1136/bmj.d686 (Published 09 February 2011) Cite this as: BMJ 2011;342:d686
  1. Philip Sedgwick, senior lecturer in medical statistics
  1. 1Section of Medical and Healthcare Education, St George’s, University of London, Tooting, London, UK
  1. p.sedgwick{at}sgul.ac.uk

Researchers undertook a meta-analysis of the analgesic effect of acupuncture.1 Randomised controlled trials of acupuncture for pain were included only if they had three arms incorporating two control groups, with patients randomised to acupuncture, placebo acupuncture, or no acupuncture. Thirteen trials were indentified. Placebo acupuncture included insertion of needles into non-acupuncture points or the use of non-penetrating needles. Separate analyses were undertaken for acupuncture versus placebo acupuncture and placebo acupuncture versus no acupuncture.

The trials used different instruments to record the primary outcome of self reported pain at the end of treatment, including visual analogue scales and ranking scales. Therefore, standardised mean differences were calculated. The results of the meta-analysis for acupuncture compared with placebo acupuncture were presented in a forest plot (figure).

Figure1

Forest plot of the effectiveness of acupuncture compared with placebo acupuncture in reducing pain at end of treatment. For each trial and the total overall effect, the mean difference in pain was derived as acupuncture minus placebo acupuncture

Which of the following statements, if any, are true?

  • a) The standardised mean difference depends on the original measurement scale

  • b) Standardised mean differences convert all outcomes to a common scale, measured in multiples of standard deviations

  • c) The sample estimates and associated 95% confidence intervals were plotted on a logarithmic scale

  • d) The overall estimated mean pain at the end of treatment with acupuncture was significantly different from that for placebo acupuncture at the 5% level of significance

Answers

Statements b and d are true, whereas a and c are false.

Last week’s question described a meta-analysis where the primary outcome was continuous and measured in the same units for each trial.2 After ascribing a weight to each trial, which was determined by the precision of its sample estimate, it was relatively straightforward to combine the results across trials to achieve an overall estimate. Acupuncture was compared with placebo acupuncture by calculating the difference between group means in pain at the end of treatment. However, the trials did not use the same method of measuring pain so a total overall estimate could not be directly obtained.

To account for differences between trials in the measurement of outcome, the mean difference between groups was standardised. A standardised mean difference was calculated as the mean difference between groups divided by the standard deviation of the outcome measure for all of the participants in the trial, regardless of group. The standardised mean difference expresses the size of the difference between treatments relative to the observed variability. It is a ratio, with both numerator and denominator in the same units as the original measurement; therefore, the standardised mean difference has no units and does not depend on the original measurement scale (a is false). The standardised mean differences convert all outcomes to a common scale, measured in multiples of standard deviations (b is true). The standardised mean differences between groups in pain were weighted before combining to obtain the overall estimate. The weighted differences are known as weighted standardised mean differences, or simply standardised mean differences.

On a forest plot the 95% confidence intervals are displayed as symmetrical around the sample estimates. Each 95% confidence interval for the population mean difference in pain between acupuncture and placebo acupuncture is symmetrical around the sample estimate. Therefore, for each trial the sample estimate and associated 95% confidence interval were plotted on a linear scale in the forest plot (c is false). In contrast the 95% confidence interval for a population relative risk is asymmetrical around the sample relative risk. When an outcome is binary and two groups are compared using a relative risk, the graphical display of the relative risks and confidence intervals are plotted on a logarithmic scale in the forest plot; the result is the confidence interval appears symmetrical around the relative risk.3

The overall estimate of the standardised mean difference between acupuncture and placebo acupuncture (acupuncture minus placebo acupuncture) was −0.17 (95% confidence interval −0.26 to −0.08). Therefore, the analgesic effect of acupuncture was greater than that of placebo acupuncture. The 95% confidence interval did not include zero—that is, a difference of no effect between acupuncture and placebo acupuncture. Therefore, pain at the end of treatment with acupuncture was significantly different from that for placebo acupuncture at the 5% level of significance (d is true). The analgesic effects of acupuncture were superior to placebo acupuncture. Because the total estimate was based on standardised mean differences it is not easy to interpret and may have limited value as a measure of treatment effect—it is not straightforward to transform back to a particular method of measuring pain. Therefore, the total estimate may serve only as a qualitative measure of strength of evidence against the null hypothesis of no difference between acupuncture and placebo acupuncture. The authors of the meta-analysis reported that the statistically significant analgesic effect of acupuncture lacked clinical relevance.

Notes

Cite this as: BMJ 2011;342:d686

Footnotes

  • Competing interests: None declared.

References

View Abstract

Log in

Log in through your institution

Subscribe

* For online subscription