Where next for evidence based healthcare?
BMJ 2013; 346 doi: https://doi.org/10.1136/bmj.f766 (Published 06 February 2013) Cite this as: BMJ 2013;346:f766All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
As the patient safety crisis at Mid Staffordshire NHS Foundation Trust has demonstrated,[1] the problems besetting healthcare extend beyond the strictly clinical. When poorly designed or executed, the actions taken to improve health services, and the systems in place for managing performance can do more harm than good. This is why evidence-based health service improvement must be the next stop for evidence-based healthcare.
Evidence-based health service improvement encompasses a variety of approaches that may go by different names, but have the same overall goal: improving the quality and safety of healthcare. Component themes include: evidence-based quality improvement,[2] the science of patient safety,[3] and evidence-based healthcare risk management.[4]
The practice of improvement has gained widespread support on the basis of its obvious good intentions, but good intentions alone have proven insufficient to the task of improving patient care. Despite some tactical successes, for instance, there is no good evidence of systems-level improvement in patient safety since the advent of the modern patient safety movement.[5] In an era of shrinking resources, in which the risks of poorly-wrought approaches to ensuring quality and safety have been made so painfully manifest,[1] it is time for the evidence-based healthcare movement to turn its eye to the problems of health service improvement.
Healthcare services are currently designed to fail. They fail their patients at an alarming rate,[5] as they fail the clinicians who are trying their best to provide safe, high-quality care. Until we build –and put to use— a robust evidence base for what works and what does not in health service improvement, we will continue wasting time, treasure, and tears without any discernable payoff in terms of patient outcomes.
References:
1. Mid Staffordshire NHS Foundation Trust. Public inquiry—chaired by Robert Francis QC. Final report. 3 volumes. 2013. http://www.midstaffspublicinquiry.com/report
2. Shojania KG, Grimshaw JM. Evidence-based quality improvement: the state of the science. Health Affairs 2005;24:138–50.
3. Shekelle P, Pronovost P, et al. Advancing the Science of Patient Safety. Annals of Internal Medicine 2011;154:693–6.
4. Card AJ, Ward JR, Clarkson PJ. Getting to Zero: Evidence-based healthcare risk management is key. Journal of Healthcare Risk Management 2012;32:20–7.
5. Landrigan CP, Parry GJ, Bones CB, et al. Temporal trends in rates of patient harm resulting from medical care. NEJM 2010;363:2124–34.
Competing interests: No competing interests
I agree that better evidence based diagnosis and also better evidence-based treatment selection is an important next step for improving health care. However, I think that the tools of evidence based medicine need to be expanded before this can be done. The concepts of sensitivity, specificity and likelihood ratio based on Bayes theorem are fine for representing screening test performance but are inappropriate for providing evidence to support clinical thinking during differential diagnosis [1].
For differential diagnosis it is the ratios of sensitivities of test results that are important and also test results that are explained in a high proportion of cases by few differential diagnoses (called good diagnostic leads) [1]. It is also important to arrive at cut-off points in an evidence-based way and not to assume in a non-evidence-based way for example that only patients with results outside two standard deviations should be included in diagnostic categories or clinical trials or treated.
Currently we only really assess therapeutic ‘efficacy’ properly in clinical trials. Instead we should also be assessing therapeutic ‘effectiveness’ in an evidence-based way at different values of test results used to select patients for treatment. This approach overcomes the problem of false positive and negative test results by imposing inappropriate cut-off points. These methods have been developed [1, 2] but are yet to be used or promoted by the evidence-based medicine community.
References
1. Llewelyn H, Ang AH, Lewis K, Abdullah A. The Oxford Handbook of Clinical Diagnosis, 2nd edition. Oxford University Press, Oxford, 2009, pp750-760.
2. Llewelyn DEH, Garcia-Puig J. How different urinary albumin excretion rates can predict progression to nephropathy and the effect of treatment in hypertensive diabetics. J Renin Angiotensin Aldosterone System. 2004; 5: 141-5.
Competing interests: No competing interests
Where next for evidence based healthcare? Surely the answer must be evidence based management.
Promoting the use of clinical evidence to improve clinical decision making is the easy part of the improvement challenge. Of course we could go on creating new evidence describing what clinicians should do, and more sophisticated ways of helping them to use it but this medical orientation ignores the bigger problem that we consistently fail to put this evidence into practice because of the suboptimal organisation and delivery of health care. This is a management challenge.
Decisions about how care should be organised and delivered are usually influenced by political pragmatism, ideology and personal experience, all guided largely by the ISLAGIATT (It Seemed Like A Good Idea At The Time) principle. Management decisions will always be influenced by less-than-rational factors but that doesn't mean that we shouldn't build a stronger evidence base to influence those decisions and more effectively utilise the large amount of social science evidence which already exists but is largely ignored. The nature of the evidence will be different from that guiding EBM, less rational, more ambiguous but nevertheless highly useful to those responsible for creating the environment within which clinical care is delivered.
Competing interests: No competing interests
The praxis of EBHC is a fully fledged reality, the fantasy is rational decision making, a notion which supports the idea that decision making can be considered separate from and outside of the ideological structures that largely determine our behaviours. The praxis of EBHC is determined by the political and economic structures resulting in much harm through overdiagnosis and overtreatment. (1,2) The editorial alludes to ‘inadequate infrastructures to protect the evidence base from avoidable bias’. The subtitle refers to evidence creators and consumers – this might seem a small point but it is the evidence based practitioners, the GPs and nurses etc that determine praxis at the coal face as it were. Evidence creation and consumption must be considered alongside praxis, and I hope the conference pays proportionate attention to its determination by a ubiquitous market logic, and the profit motive in healthcare
refs
(1) Moynihan R, Doust J, Henry D. Preventing overdiagnosis: how to stop harming the healthy. BMJ 2012;344:e3502
(2) Dempsey, O. Overdiagnosis sidelined? Breast cancer screening and The Master Discourse – ideology at work. http://www.bmj.com/content/346/bmj.f385/rr/629623
Competing interests: PhD student; Overdiagnosis and EBM - positivism ,ideology and the subject, a Lacanian/Althusserian discourse analysis
Re: Where next for evidence based healthcare?
As outlined by Drs Heneghan and Godlee (1), evidence-based medicine (EBM) in general and the Cochrane Collaboration in particular have undoubtedly contributed to improving the quality of medicine. There are a number of examples in the field of medicine in general and in neonatology/child health in particular that illustrate the importance of systematic Cochrane Reviews in improving the delivery of medical care (administration of antenatal steroids, surfactant replacement therapy, hypothermia for hypoxic ischemic encephalopathy, and probiotics to prevent necrotizing enterocolitis). Moreover, and of importance, Cochrane Reviews have also contributed to identifying interventions that are ineffective or harmful, for example, the administration of early postnatal administration of dexamethasone for neonatal respiratory distress syndrome.
However, it is important to note that EBM in general and the Cochrane Collaboration in particular are confronted with a number of problems and challenges that need to be addressed and tackled in the future. Some of these problems include:
- Improving the quality of Cochrane Review
Recent research by the Cochrane Collaboration has demonstrated that results of drug and device studies may be distorted through sponsorship by manufacturing and pharmaceutical companies (i.e. more favourable results and conclusions than sponsorship by other sources [2]. Other inventions in the field of medicine that may merit critical assessment in a RCT may not find adequate sponsoring. Therefore, adequate funding from independent organizations will be vital to tackle and solve this issue.
Moreover, a substantial percentage of systematic Cochrane Reviews still render inconclusive data and do not provide the clinician with any recommendation with regard to a specific treatment/intervention (in favour or against a certain intervention) (3, 4). Of note, the number of inconclusive Cochrane Reviews was inversely correlated to the number of included patients/studies [2]. These reviews usually conclude that, following an extensive literature search and appraisal, “insufficient trial evidence was found to guide clinical practice”. The most common reasons for inconclusive data in Cochrane Reviews are small number of patients, poor and insufficient methodology, and heterogeneous study populations (3, 4). Although defining clinical uncertainty and thereby generating new research questions is a fundamental driving force for EBM, clinicians at the bedside will find this frustrating and unhelpful. Nevertheless, by identifying important gaps in the evidence Cochrane Reviews have the potential to promote high-quality RCTs (3, 4).
However, it is also noteworthy that previous systematic reviews of Cochrane Reviews have shown that inappropriate claims of no effect or no difference may occur in about 20% of abstracts of Cochrane Reviews [5]. Conversely, it is important to mention that a recent analysis demonstrated that many apparently conclusive Cochrane Reviews may become inconclusive when the statistical analyses take into account the risk of random error due to repetitive testing [6].
- Increasing relevance to middle- and low-income countries
Furthermore, a recent synopsis of Cochrane Reviews in the field of pediatrics (neonatology and neuropediatrics) demonstrated that only a small fraction of published Cochrane Reviews originated from middle and low-income countries (5.7% in fields of neonatology and 14.3% in neuropediatrics) (7). Thus, the recommendations issued in these reviews are mostly applicable to the children in industrialized countries, which will potentially exclude the majority of children and neonates being born and cared for in the developing world. However, recently, efforts (through initiatives such as the Effective Health Care Alliance and the ‘‘Sea-orchid’’ consortium) have been undertaken to disseminate knowledge from the Cochrane Review Group to low- and middle-income countries to ensure that care practices are evidence-based and that scarce resources are used and allocated appropriately.
- Keeping reviews up to date
Although it is of great importance that Cochrane Reviews are kept up-to-date (i.e. they should be updated every 2 years), in reality this aim is difficult to realize. In two recent systematic analyses of Cochrane reviews a substantial proportion of reviews was not kept up-to-date (3, 4). Thus, it will be necessary to prioritize reviews for updating based on perceived clinical importance and knowledge of availability of new evidence (8).
In summary, the Cochrane Collaboration has been now at the forefront of promoting and implementing principles of EBM into clinical practice for two decades as outlined by Drs Heneghan and Godlee [1]. Apart from organizational, political and administrative efforts that need to be realized by the Cochrane Collaboration and by other institutions strongly affiliated with evidence based healthcare, it will be important that the medical research community (yes, that is us!) will plan, organize, and participate in high-quality RCTs that can serve as the basis for implementing EBM at the bedside. To cut down on the number of inconclusive Cochrane meta-analyses, it will be of paramount importance to include adequate numbers of study patients. This challenging endeavour will best be realized through good networking and cooperation of the medical community worldwide, irrespective of geographic origin (low- and middle-income countries or highly-industrialized Western world). When successfully tackling the above mentioned issues and obstacles, the already amazing success story of EBM and the Cochrane Collaboration will even become more substantial.
References
1. Heneghan C, Godlee F. Where next for evidence based healthcare? BMJ. 2013 Feb 6;346:f766. doi: 10.1136/bmj.f766.
2. Atwood K. The ongoing problem with the National Center for Complementary and Alternative Medicine. Skeptical Inquirer 2003; 27:3–11
3. Willhelm C, Girisch W, Gottschling S, Gräber S, Wahl H, Meyer S. Systematic Cochrane Reviews in Neonatology: A Critical Appraisal. Pediatr Neonatol. 2013 Apr 18. pii: S1875-9572(13)00037-5. doi: 10.1016/j.pedneo.2013.03.002. [Epub ahead of print]
4. Meyer S, Gortner L, Larsen A, Kutschke G, Gottschling S, Gräber S, Schroeder N. Complementary and alternative medicine in paediatrics: a systematic overview/synthesis of Cochrane Collaboration reviews. Swss Med Wkly 2013; 143:w13794
5. Alderson P, Chalmers I. Survey of claims of no effect in abstracts of Cochrane reviews. BMJ 2003; 326:475
6. Brok J, Thorlund K, Wetterslev J, Gluud C Apparently conclusive meta-analyses may be inconclusive--Trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses. Int J Epidemiol. 2009 Feb;38(1):287-98
7. Sascha Meyer, Christiane Willhelm, Wolfgang Girisch, Sven Gottschling, Stefan Gräber and Ludwig Gortner. The Role of Developing Countries in Generating Cochrane Meta-analyses in the Field of Pediatrics (Neonatology and Neuropediatrics): A Systematic Analysis. World Health & Population, 14(2) March 2013: 24-32.doi:10.12927.whp.2013.23272
8. McGuire W, Fowlie PW, Soll RF. What has the Cochrane collaboration ever done for newborn infants? Arch Dis Child Fetal Neonatal Ed. 2010 Jan;95(1):F2-6. doi: 10.1136/adc.2007.133561. Epub 2009 May 3.
Competing interests: No competing interests