Choosing the best research design for each questionBMJ 1997; 315 doi: https://doi.org/10.1136/bmj.315.7123.1636 (Published 20 December 1997) Cite this as: BMJ 1997;315:1636
It's time to stop squabbling over the “best” methods
Lots of intellectual and emotional energy, ink, paper, and readers' precious time have been expended comparing, contrasting, attacking, and defending randomised control trials, outcomes research, qualitative research, and related research methods. This has mostly been a waste of time and effort, and most of the disputants, by focusing on methods rather than questions, have been arguing about the wrong things.
Our thesis is short: the question being asked determines the appropriate research architecture, strategy, and tactics to be used—not tradition, authority, experts, paradigms, or schools of thought.
If the question is, “What is the importance of patient preferences in the choice of treatment for benign prostatic hyperplasia?” the appropriate study architecture, strategy, and tactics are those that identify and characterise the reactions of individual patients to their disease and their assessments of the risks and benefits of alternative treatments through open ended, in depth interviews (to the point of redundancy or saturation), with emphasis on variations in preferences among individuals. The fact that this array of approaches is called qualitative research is irrelevant to whether this is the best way to answer this question.
If the question is, “In men with benign prostatic hyperplasia is laser prostatectomy superior to transurethral resection of the prostate in terms of symptom relief, blood loss, and the length of catheterisation and hospital stay?” the appropriate study architecture, strategy, and tactics are those that assemble a group of individuals with this condition, randomise them (concealing the assignment code) to the alternative procedures, and achieve complete follow up of their subsequent outcomes. The fact that this combination of approaches is called a randomised control trial or efficacy research is irrelevant. Because it minimises the confounding of treatment and prognosis, a trial is the best way to answer questions of this sort (especially when several trials are combined into a systematic review or meta-analysis).
If the question is, “Are we providing effective care to patients with benign prostatic hyperplasia in our region, and are they appearing to benefit from it?” the appropriate study architecture, strategy, and tactics are those that succeed in assembling and describing patients with benign prostatic hyperplasia in a specified population, describing the interventions they receive and events they experience, and completing follow up to the ends of their lives or the study period, whichever is later. Variations in the rates with which they receive interventions shown in randomised trials to do more good than harm answers the first part of the question. (For interventions where randomised clinical trials have not been performed, the variations in treatment rates obtained by studies of the course of the disease may help create the sense of uncertainty that allows a randomised clinical trial to be initiated.) Disparities between interventions and outcomes or between the treatment patients receive and the treatment they prefer answer the second part and raise a further series of questions about why that might occur. The fact that this array of approaches is called non-experimental cohort study, outcomes research, or effectiveness research is irrelevant: these happen to be the appropriate methods for answering these sorts of questions.
The answers provided to each of these questions by the architectures we have suggested could in themselves generate questions whose answering requires a shift to another research method. Furthermore, all three questions could be addressed using other architectures, strategies, and tactics (including the solicitation of “expert” opinion) but, we suggest, not as well. Finally, we could try to answer them all with data already gathered for some other purpose.
Each method should flourish, because each has features that overcome the limitations of the others when confronted with questions they cannot reliably answer. Randomised controlled trials carried out in specialised units by expert care givers, designed to determine whether an intervention does more good than harm under ideal conditions, cannot tell us how experimental treatments will fare in general use, nor can they identify rare side effects. Non-experimental epidemiology can fill that gap. Similarly, because the theoretical concerns about the confounding of treatment with prognosis have been repeatedly confirmed in empirical studies (in which patients who accept placebo treatments fare better than those who reject them), non-experimental epidemiology cannot reliably distinguish false positive from true positive conclusions about efficacy. Randomised trials minimise the possibility of such error. And neither randomised trials nor non-experimental epidemiology are the best source of data on individuals' values and experiences in health care; qualitative research is essential.
But focusing on the shortcomings of somebody else's research approach misses the point. The argument is not about the inherent value of the different approaches and the worthiness of the investigators who use them. The issue is which way of answering the specific question before us provides the most valid, useful answer. Health and health care would be better served if investigators redirected the energy they currently expend bashing the research approaches they don't use into increasing the validity, power, and productivity of ones they do.