Evidence based medicine: flawed system but still the best we’ve gotBMJ 2014; 348 doi: https://doi.org/10.1136/bmj.g440 (Published 22 January 2014) Cite this as: BMJ 2014;348:g440
All rapid responses
I want to formally reject the null hypothesis of my research.(1)
I respectfully disagree with the statement about EBM in the title of this article: "flawed system but still the best we've got."
I think is quite the opposite: it is probably the worst we've got.(2)
You mentioned Winston Churchill in this paper and I would like to quote him: "We shall not surrender."
The strength of evidence has been frequently represented as a pyramid: there is no pyramid of evidence - it is a collective delusion and now medicine has been severely broken.
We must fix medicine now, but first we must recognize that EBM is not evidence-based and it is causing medicine a great damage. We need to stop this nonsense now before it's too late.
It was already predicted by one of the most cited EBM (position 28th out of 9.476 articles) articles on ISI Web of Knowledge (2001 - 2014).
"If treatment decisions are made on the basis of misleading methodological tests, the costs to patients and society could be high." (3)
The most cited articles regarding EBM are:
1st. Surviving sepsis guidelines: 2008 (2303 citations)
2nd. Surviving sepsis guidelines: 2004 (1571 citations)
3rd. to 50th. Available on reference number 4.
This is also evidence that citations do not reflect the quality of an article, as well as the impact factor does not accurately measure the reputation of a peer-review journal.
I will submit my research findings hoping to receive a strict prepublication and postpublication peer review according to the best available evidence.(5)
1. Hierarchical levels of evidence based medicine (EBM) are wrong.
2. Is the pharmaceutical industry like the mafia? Yes. http://blogs.bmj.com/bmj/2013/09/10/richard-smith-is-the-pharmaceutical-...
3. Lau J, Ioannidis JPA, Terrin N, et al. The case of the misleading funnel plot. BMJ 2006;333:597–600. doi:10.1136/bmj.333.7568.597
4. Chaos theory to evidence-based medicine: Hello!
URL: http://wp.me/p4UVo9-5z (date accessed: Dec 17, 2014).
5. Evidence based publishing. URL: http://www.bmj.com/about-bmj/evidence-based-publishing (date accessed: Dec 17, 2014).
Competing interests: Non-financial conflict of interests:
Evidence based medicine relies on dangerous assumptions that are not universally valid. The Bayesian approach mentioned is as fraught with danger as is the Frequentist approach, and Survival Analysis is far from the statistics used in controlled experiments. We are using tools without having the understanding of the limitations so say things that are for the most part arbitrary, and fundamentally marketing popularity and fad rather than fact. Individual variation for patients makes the use of large group averages problematic. Averages of ratios and rates (ie, Hazard Ratios) is also valid only in special cases, and presents idiosyncratic and biased findings as objective, which corrupts the very basis of medical decision making. See http://www.biomedcentral.com/1471-2288/13/152.
Competing interests: No competing interests
It was interesting to read this article.
In addition to the various limitations of EBM discussed, the applicability of the evidence base in developing countries is sometimes debatable.
Much of the evidence base in current medical literature is based on studies conducted in developed countries and published in reputed journals. Though the search for literature needs to be exhaustive in systematic reviews/meta-analyses, the bias of data representativeness remains and the evidence is not equally applicable to all populations.
Apart from difference in biological/racial/ethnic variables, social economic and environmental variables affect medical decision making. The availability, accessibility and cost of care is one of the important variables in deciding the type of treatment/intervention. For example, in the management of early stages of cancers in developed countries, conservative management and regular follow up may be the intervention of choice since patients have high awareness regarding health, health care is available and accessible and the patient is likely to return for review and follow-up. However, in rural and tribal areas in developing countries, where the patient has very poor health awareness and health care is often not easily available and accessible, it may be more appropriate to think of more radical approaches since the patient is unlikely to adhere to a follow up protocol and that visit may be the only visit to a health facility before the patient returns much later with widespread terminal cancer or the patient may not return at all.
Competing interests: No competing interests
Unfortunately, evidence-based medicine is not uniformly triumphant.
Consider Community Treatment Orders (CTOs) in psychiatry. These, since 2008, have imposed anti-psychotic mediction on pateints, even outside mental hospitals. CTOs impinge on human rights.
The evidence base seems to be against CTOs. The latest trial by Burns in the Lancet (1) concluded that CTOs were not effective above control, yet clinicians remain perhaps overenthusiastic about them (2).
The evidence base on CTOs is indeed being ignored; 'clinical impressions unsubstantiated by evidence cannot be sufficient to justify CTOs' (3).
CTOs provide a maybe disturbing prism through which we witness how psychiatry, and really medicine as a whole, are not so free of unscientific bias as they pretend to be.
Tabloid pressure for CTOs, the social stigma punishing schizophrenics, must have swayed the outlook of clinicians.
Medicine in general is influenced by the greed of Big Pharma, notably with anti- psychotic medications from chlorpromazine to clozapine, as painfully detailed by Robert Whitaker (4).
Stepping outside the boundaries of evidence-based medicine is to enter the darkness.
(1) Community treatment orders for patients with psychosis (OCTET): a randomised controlled trail. Burns T et al. Lancet 2013; 381: 1627-33.
(2) Compulsion in the community: mental health professionals' views and experiences of CTOs. Coyle D et al. The Psychiatrist (2013), 37, 315-321.
(3) Can we reverse the rising tide of compulsory admissions? Lancet Comment. 381:1603-1604.
(4) Mad in America. Robert Whitaker. Perseus Books. 2002. Passim.
Competing interests: No competing interests
The editorial comment  and the thought-provoking article by Des Spence  on EBM seem to have provoked a number of responses, but almost all relate to medicine, pharmacological industry and medical politics. Is the language of ‘icon’ and ‘iconoclasts’  is reminiscent of a phenomenon to be worshipped with its pronouncements as ‘commandments’ not to be questioned? Surely not! As an EBM enthusiast, the author acknowledges its special significance in surgery but also very peculiar flaws and limitations.
Peculiarities of surgical RCTs: Firstly, there are major differences in the trials comparing two drugs as opposed two different variations of a surgical technique because of different skills and ingrained practices of surgeons. A good example is the comparison between laparoscopic ‘salpingotomy’ and ‘salpingectomy’ for tubal ectopic pregnancy . Randomised controlled trials (RCT) would tend to condense the results (e.g. future intrauterine and ectopic pregnancy rates) towards an average and may show outcomes of ‘salpingotomy’ similar (or even inferior) to ‘salpingectomy’. Should the guideline-makers conclude that even the skilled surgeons achieving results towards the top end (reported in many case series) should stop performing ‘salpingotomy’ or this operation be condemned and not taught to the juniors?
Secondly the EBM is only as good as the evidence and the people making decisions  (i.e. the guideline-makers and surgeons this context). In the presence of an equipoise, an organ conserving surgery (salpingotomy) should have been a default choice rather than an extirpative one (technically easier salpingectomy) especially when preferred by patients given informed choice. The debateable ‘evidence based’ guidance favouring salpingectomy became popular and entrenched because odds were already stacked against ‘salpingotomy’ with a longer learning curve and increased need for senior supervision. The revised guidelines of 2004  do propose ‘salpingotomy’ as a valid option but in a confusing negatively phrased recommendation – “In the presence of a healthy contralateral tube there is no clear evidence that salpingotomy should be used in preference to salpingectomy”. This statement is almost universally interpreted even by senior clinicians as ‘(based on EBM) salpingectomy is preferable to salpingotomy’, despite the description that follows alludes to studies showing some limited evidence of benefits of salpigotomy over salpingectomy . As a result of misinterpretation and misrepresentation of EBM, ‘salpingotomy’ is unlikely to recover its justified position in the management of ectopic pregnancy.
A Bayesian perspective:
The most commonly performed major surgical procedure is ‘Caesarean section’ (CS). Numerous studies and RCTs on different aspects of surgical techniques of CS, mostly relating to short term outcomes , show completely different, contradictory and heterogeneous results including the CORONIS multinational RCT on 15935 women showing all variations of techniques acceptable . This of course highlights the much bigger element of ‘Bayesian’ approach in formulating EBM in surgical practice. The surgeons should use their prior beliefs based on their own experience, observation, reasoning and theoretical principles in addition to the trial data to choose a particular technique (even if in variance with some ‘guidelines’). NICE routinely reminds that their guidance should not replace professional judgment. A particularly interesting example is ‘sharp’ (Pfannensteil) versus ‘blunt’ (Joel-Cohen) abdominal entry techniques for CS. Almost all junior Obstetricians have adopted ‘blunt’ abdominal entry technique based on purported EBM.
Several studies have given variable results, but a 2008 and 2013 Cochrane review  constituting two studies (310 and 101 women) is taken as proof of superiority of ‘blunt’ entry over the ‘sharp’ one (refuted by the CORONIS trial). However, this review suffers from drawbacks outlined by Des Spence  namely nonspecific surrogate ends (e. g. febrile morbidity, pain), clinically irrelevant outcomes (minute savings of operating time or blood loss) and conflicts of interest. Pfannensteil incision involve both sharp as well as blunt dissection. The current practice of ‘blunt entry’ involves making an opening in the abdomen of 10 x 10cm at least almost entirely by stretching (blunt force) after skin incision . This is against good surgical principles because it is a relatively common observation that often excessive force is applied leading to blunt trauma to tissues beyond the incision, bruising, haematomas and increased incidence of wound infection (spreading cellulitis in some cases). Moreover, the habit of rough handling of tissues then extends to other parts of surgical procedure. Senior surgeons who have achieved near zero incidence of wound complications over a couple of decades with their Pfannensteil type abdominal entry could not be expected to switch over to blunt entry. Moreover, there is a continuum between a ‘sharp’ and ‘blunt’ entry technique. It is also overly simplistic to assume that surgeons who have formed a habit of applying significant blunt force will suddenly stop using it when asked to perform ‘sharp entry’ technique in an RCT.
Another recommendation by national guidelines ‘not to suture visceral and parietal peritoneum’ during CS is very widely practised. This recommendation could be argued to be derived from debatable EBM with drawbacks  of short term unrelated surrogate ends (postoperative pain) and clinically irrelevant outcomes (minute time saving). On the other hand many surgeons have noticed a much higher incidence of adhesion formation (sometimes extensive adhesion of uterus to abdominal wall) when peritoneum is not sutured with long term morbidity . It’s high time this recommendation is reversed especially in the light of two recent large RCTs .
In summary, the purported ‘EBM’ or ‘guidelines’ have a hypnotic effect on junior doctors. Misjudged acquisition of injudicious surgical techniques will have long term impact on surgeons which cannot be quickly reversed like changing drug therapy when evidence changes. Even senior doctors despite their scepticism about some of the ‘EBM based guidelines’ feel pressurised or find it inexpedient to express their views. There has also been increasing attempt to micromanage surgical craft by minor opinion-based recommendations like non-suturing of subcutaneous tissue or placing laparoscopic incision inside the umbilicus etc. Surgeons should be encouraged to think independently and develop their skills based on experience and information about pros and cons, in the knowledge of EBM and its limitations, rather than subjected to commandments.
Mr Shashikant L SHOLAPURKAR, MD, DNB, MRCOG
1. Godlee F. Evidence based medicine: flawed system but still the best we’ve got. BMJ 2014;348:g440
2. Spence D. Evidence based medicine is broken. BMJ 2014;348:g22.
3. Sholapurkar SL. Salpingotomy Vs Salpingectomy for Ectopic Pregnancy and Evidence Based Medicine. Diagnosis and management of ectopic pregnancy Rapid Response. BMJ 21 June 2011 (http://www.bmj.com/rapid-response/2011/11/03/salpingotomy-vs-salpingecto...)
4. Royal College of Obstetricians and Gynaecologists. The management of tubal pregnancy. Guideline 21. 2004. www.rcog.org.uk/womens- health/clinical-guidance/management-tubal-pregnancy-21-may-2004.
5. Sholapurkar SL. Increased incidence of placenta praevia and accreta with previous caesareans--a hypothesis for causation. J Obstet Gynaecol 2013;33:806-9.
6. CORONIS Collaborative Group, Abalos E, Addo V, Brocklehurst P, El Sheikh M, Farrell B, Gray S, Hardy P, Juszczak E, Mathews JE, Masood SN, Oyarzun E, Oyieke J, Sharma JB, Spark P. Caesarean section surgical techniques (CORONIS): a fractional, factorial, unmasked, randomised controlled trial. Lancet. 2013;382:234-48.
7. Mathai M, Hofmeyr GJ, Mathai NE. Abdominal surgical incisions for caesarean section. Cochrane Database Syst Rev. 2013 May 31;5:CD004453. doi:10.1002/14651858.CD004453.pub3.
Competing interests: No competing interests
Evidence-based medicine’ (‘EBM’) as currently proclaimed takes a narrow view of ‘evidence’ and ‘medicine’, focussing mainly on assessing and summarising treatment efficacy and the specificity and sensitivity of dichotomised test results based on non-evidence based ‘gold standards’. EBM needs to have wider horizons if it is to support better those who guide people through their illnesses and other health concerns.
The term ‘EBM’ could also include the ‘patient’s personal evidence’ by specifying in each record the symptoms, personal values, signs and test results that were used for each diagnosis and to choose each treatment, in order to promote transparency, continuity of care, reduction of errors and better research [1,2]. The term ‘EBM’ should also include research evidence on differential diagnosis based on probabilistic reasoning by elimination [2,3,4] and diagnostic and treatment selection criteria based on individual measurement results [2,4,5,6]. This is what will be needed for what Sir Muir Gray in the ‘Oral History’ video calls personal, stratified medicine, which he predicts to be the future .
An inability to base diagnostic and treatment criteria on sensible evidence will inevitably result in the pharmaceutical industry creating over-diagnosis and over-treatment. It is not all be due to profit motive and conflict of interest in guideline writers; there is also poor know-how. The research community, journals and regulators (not just proclaimers of ‘EBM’) must not think that the current idea of EBM is the best we’ve got but work to widen its horizons if they are to avoid continued disillusion.
1. Llewelyn H. How to promote and support a culture of patient-centred openness and transparency in the NHS. International Journal of Clinical Practice Volume 67, Issue 10, pages 943–946, October 2013
2. Llewelyn H, Ang AH, Lewis K, Abdullah A. The Oxford handbook of clinical diagnosis. 2nd ed. Oxford University Press, 2009:749-72.
3. Llewelyn, D E H. (1979) Mathematical analysis of the diagnostic relevance of clinical findings. Clin. Sci., 57, 5, 477-479.
4. Llewelyn H. Assessing properly the usefulness of clinical prediction rules and tests. BMJ 2012;344:e1238.
5. Llewelyn DEH, Garcia-Puig J. How different urinary albumin excretion rates can predict progression to nephropathy and the effect of treatment in hypertensive diabetics. J Renin Angiotensin Aldosterone Syst 2004; 5:141-5.
6. Llewelyn H. Analysis of clinical trial data by using evidence based triage reduces over-diagnosis. Preventing over-diagnosis conference, Dartmouth College, New Hampshire, 2013 –abstract #11. http://preventingoverdiagnosis.net/documents/POD-Abstracts.docx
7. Smith R, Rennie D. Evidence based medicine—an oral history. BMJ 2014;348:g371. Video: http://ebm.jamanetwork.com
Competing interests: No competing interests
“Facts do not cease to exist because they are ignored.” Aldous Huxley.
I must congratulate you for finding this out now. I am giving here an abstract of an article that I wrote a couple of decades ago.
EBM should be a concerted effort to retrieve and synthesize as much data as is available to make it possible for practising physicians to incorporate in their practice. The conventional clinical acumen, intuition and clinical experience are deprecated in this process. I wonder if we have enough data to guide us all the time in clinical practice for the benefit of patients. One glaring example is the area of hypertension treatment. Whereas there are more than six guidelines in the world for doctors to follow, if all of them are computed together, the inclusion criteria would add up to only 39% of patients. The majority of 61% patients do not have guidelines. Similarly, there are as many reviews eulogizing coronary interventions in the immediate post infarction period as there are which show them in bad light. (1)
Almost all the randomized studies cited in the reviews written, whether from the rich or poor countries, relate to ideal patients with a single intervention at a time, while in practice in real life situations rarely does one encounter patients in such an ideal slot. The practising doctors might have to innovate. More than all that is the bane of poly-pharmacy that doctors have to, per force, practise in many clinical situations. Drug interactions are a big problem and the resulting adverse drug reactions (ADR) rank fourth as the important cause of death in the US. (2) What, then, is the basis of EBM authority? There is a nexus between the EBM leaders and the industry. We depend on those guidelines for our EBM!
“Recently Senator Charles Grassley, ranking Republican on the Senate Finance Committee, has been looking into financial ties between the pharmaceutical industry and the academic physicians who largely determine the market value of prescription drugs. He hasn’t had to look very hard. Perhaps the most egregious case exposed so far by Senator Grassley is that of Dr. Charles B. Nemeroff, chair of Emory University’s department of psychiatry and, along with Schatzberg, coeditor of the influential Textbook of Psychopharmacology. Nemeroff was the principal investigator on a five-year $3.95 million National Institute of Mental Health grant—of which $1.35 million went to Emory for overhead—to study several drugs made by GlaxoSmithKline. To comply with university and government regulations, he was required to disclose to Emory income from GlaxoSmithKline, and Emory was required to report amounts over $10,000 per year to the National Institutes of Health, along with assurances that the conflict of interest would be managed or eliminated.”( Drug Companies & Doctors: A Story of Corruption by Marcia Angell, January 15th 2009 in the New York Review of books. (3)
Modern medicine has become mandatory for emergency care, anyway. On a long term basis, especially in chronic diseases, most of our interventions have been shown in very poor light. (4) The quick-fix medical care interventions described above are for the minority that is ill and not for the well majority. The present effort to use medical care methods for all results in much misery and iatrogenic problems. (2) Doctors’ strike in Israel recently and, on a couple of earlier occasions in the distant past, did show that when medical interventions came down death rate in the population fell down significantly and vice-versa. (5)
I am glad that even Sir Michael Rawlins did hint that RCTs have been undeservedly put on a high pedestal in his Oration at the RCP London in 2008. RCTs have very little science base. Nomenclature and guideline writers have come under the cloud lately. Readers would do well to read more about this in the “ADHD-Leon Eisenberg” saga which came to light lately. (6) One can feel butterflies in the stomach reading the latest hypertension guidelines from the Joint National Committee of the USA chaired by Dr. Marvin Moser. (7) A more liberal levels have been suggested there. Those were the levels that I was taught for diagnosing hypertension when I was a student in 1957. We have come a full circle. Julian Tudor Hart had come to similar conclusions four decades ago through his meticulously excellent record keeping of his cohort of mining community in North Wales.(8)
1) Hegde BM. To do or not to do-Doctor’s dilemma Kuwait. Med. J 2001; 33(2): 107-110.
2) Starfield B. Is US medicine the best in the world? JAMA 2000; 284: 483-485.
4) McCormack J and Greenhalgh T. seeing what you want to see in randomized controlled trials: versions and perversions of UKPDS data. BMJ 2000; 320: 1720-23.
5) Siegel-Itzkovich J. Doctors’ strike in Israel may be good for society. BMJ 2000; 320: 1561.
6) Eisenberg L. www.snopes.com › Home › Politics › Quotes
7) Moser M. www.upi.com › Health News and www.ash-us.org › Scientific Meetings
8) Tudor Hart J. www.goodreads.com/author/show/466802.Julian_Tudor_Hart
Competing interests: No competing interests