Observational studies should carry a health warningBMJ 2007; 334 doi: https://doi.org/10.1136/bmj.334.7586.179-c (Published 25 January 2007) Cite this as: BMJ 2007;334:179
Observational studies have their place, although the results often depend crucially on the type of analysis used to generate them. A good illustration of this principle comes from a study comparing four different ways of looking at the effects of invasive revascularisation after heart attack. Essentially, all four methods adjusted for the many baseline differences between people who have invasive treatments and people who don't, differences that would normally be eliminated by randomisation in a randomised trial.
Using data from 73 238 Medicare patients, the authors showed that standard analytical methods—multivariate risk adjustment and two methods based on propensity scoring—came up with a survival benefit of around 50% (adjusted relative risks 0.51, 0.54, and 0.54). A newer method called instrumental variable analysis indicated a more modest survival benefit of 16% (adjusted relative risk 0.84), closer to the results from randomised trials. Does the new analysis get nearer “the truth” than other methods?
Not necessarily, says a linked editorial (pp 314-6). Estimating treatment effects from observational studies will never be an exact science. The best researchers can do is make sure their analysis includes all the important baseline variables and balances them between treated and untreated groups. The best we can do is to interpret any results with caution.