David Oliver: Should practical quality improvement have parity of esteem with evidence based medicine?
BMJ 2017; 357 doi: https://doi.org/10.1136/bmj.j2582 (Published 30 May 2017) Cite this as: BMJ 2017;357:j2582
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
I thank Dr Welch for her comments and agree with them
I would say that there are accepted formats for conducting and reporting quality improvement work. Not only are the methodological approaches field tested, standardised and publicly available but so are consensus guidelines for reporting such as SQUIRE and STaRI
It just so happens that most good Quality Improvement work ends up either published as case studies by organisations like the Health Foundation or in bespoke journals with lower impact factor and therefore less weight in conventional academia, such as BMJ Safety and Quality
As some of the other responses to my column suggest, this needs to change. While we argue through peer review about the methodological rigour and construction of clinical trials and seek perfect evidence, we have improvement science on the ground in health services, being used for groups of patients in all their complexity without carefully constructed exclusions and ever changing workforce and policy environment in all theirs
The paradigm in which QI (or action research, or policy analysis etc) are seen as inferior sources of evidence is one constructed by the EBM movement and therefore seen through its own prism
David Oliver
Competing interests: No competing interests
Thank you, David, for another realistic, doable and eminently sensible viewpoint.
Whilst the EBM revolution has reshaped Medicine (and Nursing and AHP) into knowledge driven, scientifically (sometimes pseudo-scientifically) professions, it fails to address the multi-factorial reality of what actually happens to the patients, the 'Art' of helping humans. The science of EBM is largely reliant on consistent, reproducible conditions, but humans are not consistent, reproducible creatures, be they patients or practitioners.
Of course EBM must have a place in learning and testing new knowledge, but alone it cannot ensure that patients benefit from the correct Care in their individual circumstances. EMB is just too starchy to cope with the inevitable twists and turns of life, and the individual quirks we all have. Nor can it effectively address the many non-technical factors that will influence outcomes- staffing, working cultures, education, environment. Hence the value of QI, which by its own nature (when done properly) is all about looking at the factors that most affect journeys and outcomes locally with flexibility. But whilst journals continue to demand strictly scientific criteria, and in the absence of accepted formats for data collection and publication, QI and patient safety will continue to be provincial under dogs to the mighty Academic Evidence Base.
Competing interests: No competing interests
Agree 100%. In fact, I would even suggest that practical QI to implement the vast ocean of well established evidence should outweigh new RCTs chasing marginal gains.
There is no shortage of very well established evidence based guidelines in existence, majority of which are inconsistently implemented (eg CT head rules) - there will be an exponentially greater potential clinical impact if we valued formal QI methodologies to bring them to the bedside, than investing further in RCTs investigating tiny gains (and which will likely end up not being implemented anyways)!
Academic recognition and investment in QI should reflect that...
Competing interests: No competing interests
This Acute Perspective by Dr David Oliver (1) has our interest, in part because we all embarked on our career in medicine around the same time.
We very much share Dr Oliver's advocacy for "the actions and engagement of frontline practitioners and the real world context in which they work" and agree that these "are critical to success."
We would like to contribute in the spirit of critical thinking (2) regarding the place of ethics in Quality Improvement (QI).
Dr Oliver states that QI can deliver “tangible outcomes” and that it has “a methodological and theoretical rigour and peer community of its own”.
As far back as 2007 Brent et al identified that "ethical issues arise in QI because attempts to improve quality may inadvertently cause harm, waste scarce resources, or affect some patients unfairly."(3)
Dr Oliver states that “ethical approval is less burdensome” for QI. We are of the view that ethics must be one of the necessary starting principles for any QI work and would argue that any attempt, however well intentioned, to demote ethics from this role might result in outcomes that may not be described as “improvement”.
References:
(1) Oliver, D. Acute Perspective: Should practical quality improvement have parity of esteem with evidence based medicine? Published 30 May 2017. BMJ 2017;357:j2582
(2) Sharples et al. Critical thinking in healthcare and education. Published 16 May 2017. BMJ 2017;357:j2234
(3) Brent, J et al. The Ethics of Using Quality Improvement Methods in Health Care. Ann Intern Med. 2007;146:666-673. https://qi.elft.nhs.uk/wp-content/uploads/2017/02/Ethics-of-QI.pdf
Competing interests: No competing interests
Suppressing my Canadian geriatrician’s instinct to see both sides of any argument, I write to side with well-organized Quality Improvement initiatives.[1] QI studies commonly elicit reflex rubbishing in evidence-based, checklist-wielding reviews. This is unfair because health care providers must get to grips with the needs of people who get ill. Such people include those (typically older and frail) who, in virtue of the complexity of their illnesses, receive short shrift in an evidence base mostly rooted in single-system illnesses.[2] Standard teaching (GRADE being perhaps less guilty than other schemes [3]) heavily judges a study’s value by its design. This is especially unhelpful when the same reviews simultaneously ignore inclusion/exclusion criteria that systematically exclude frail patients from randomized, controlled trials, a luxury not afforded all-comer QI studies.
This everyday disparity offers opportunity. The information value of QI projects is enhanced when their results are set in the context of what is known from clinical trials. Do their results align with the outcomes reported by the trials? If not, is deviation informative? QI projects taking place at multiple sites using a common core protocol - perhaps based on a first successful implementation, as is done by the Acute Frailty Network [4] - can be a vital means of improving care, especially that of frail older adults with cognitive and/or sensory impairment, who live alone, against whom many standard procedures are biased.
Aggregating the large body of QI that now goes on to make it more widely available is likely to be useful to policy makers.[5] It is also one way to shake off the shackles of single system thinking in time to offer useful evidence for the care of patients with complex needs.
References:
1. Oliver D. Should practical quality improvement have parity of esteem with evidence based medicine? BMJ 2017;357:j2582
2. Tisminetzky M, Bayliss EA, Magaziner JS, et al., Research priorities to advance the health and health care of older adults with multiple chronic conditions. J Am Geriatr Soc. 2017 May 26. doi: 10.1111/jgs.14943. [Epub ahead of print]
3. Alonso-Coello P, Oxman AD, Moberg J, et al., GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. BMJ. 2016;353:i2089. doi: 10.1136/bmj.i2089.
4. Conroy S. The Acute Frailty Network. https://britishgeriatricssociety.wordpress.com/2015/03/19/the-acute-frai... Accessed June 3, 2017
5. Whitty CJ. What makes an academic paper useful for health policy? BMC Med. 2015 Dec 17;13:301. doi: 10.1186/s12916-015-0544-8.
Competing interests: No competing interests
Great piece
Absolutely agree.
I'd say the need for parity becomes more important as the evidence matures, where we get closer to the top of the curve of diminishing marginal returns evidence wise; and where we get to the point where we have enough knowledge to act, but maybe not perfect knowledge. That action should go with ongoing evaluation and a feedback loop.
We fail to capitalise on potential (often quite significant) gain by our continued failure to put knowledge consistently into practice.
This failure can result in undertreatment and over treatment, as has been demonstrated in NHS Atlases, National Clinical Audits and similar year after year. For instance how many more National Diabetes Audits do we need to tell us that we undermanage risk in many and over treat many with poorly evidenced A1C targets, this is the failure of over reliance on the mindset that perfect evidence leads to change. It causes harm, wastes resources and leads to lost opportunities for benefit.
However, as well as recognising there is an issue, we need to be clear in our strategies for upskilling our workforce around QI methodologies. There's no real rocket science for this, and the are readily learned and accessible to all. We do need to invest time into this, it won't happen by magic.
The alternative is perhaps we go on forever creating ever more perfect evidence base that doesn't lead to any change.
Competing interests: No competing interests
Re: David Oliver: Should practical quality improvement have parity of esteem with evidence based medicine?
Thank you for this defence of quality improvement, which as you point out, has its own paradigm, and own science, drawing on the scientific method, an understanding of variation grounded in its own statistical theory and the fields of human behaviour and psychology. When undertaking quality improvement work, practitioners are sometimes asked questions about sampling and p values by clinicians who are hoping to see a randomised controlled trial rather than a quality improvement work. If you want to conduct an RCT, go for it. But if you're trying to reduce falls or waiting times, you are not trying to compare two matched groups of patients. You are trying to improve performance over time, which, in the real world, means comparing different patients and different staff.
You can't control for that. But you can come up with explicit theories for change, based upon sound hypotheses, test these changes and identify whether these changes have had an effect by identifying whether variation is common cause or special cause. This is based in a theory which is just as valid as the randomised control paradigm, just different. But for clinicians grounded in research it feels alien, unfamiliar and thus must somehow not be as good as the "gold standard."
It's time to move away from this. The evidence based movement practitioners and quality improvement practitioners both have their complementary roles. Whilst high quality research can greatly advance knowledge and inform what works in perfectly controlled circumstances, quality improvement seeks to learn what works here, now, with these people and these resources. We don't do it justice by asking questions of it that are suited to research and not improvement.
This is a scenario akin to a grizzled butcher going into a bakers and asking for a pound of bread. In a bakery, there is nothing wrong with buying by the slice or loaf.
Competing interests: No competing interests