Why we can’t trust clinical guidelinesBMJ 2013; 346 doi: https://doi.org/10.1136/bmj.f3830 (Published 14 June 2013) Cite this as: BMJ 2013;346:f3830
All rapid responses
We thank Dr Gordon for drawing our attention to this issue, which we have discussed previously with him.
We accept that some of our legacy guidelines were produced under different governance arrangement to the current process. All interests, however, were declared and managed during development, and were available on publication of all of the extant guidelines to which Dr Gordon refers and for three years thereafter.
Under our current process a register of interests for the guideline development group members can be viewed online on the SIGN website.
Competing interests: No competing interests
Jeanne Lenzer’s feature article, “Why we can’t trust clinical guidelines(1)” has attracted a considerable number of responses, many of which highlight individual clinical guidelines where particular issues of conflicts of interest have arisen.
One course of action would be that any clinical guideline should have all potential conflicts of interest recorded and as easily available to the reader as the guideline is. This would allow the reader to make his/her own assessment of the advice given. Indeed the Scottish Intercollegiate Guidelines Network (SIGN) states that “SIGN is committed to open declaration of competing interests in all its activities(2).” However, the Programme Lead for the SIGN stated on the 20th November 2013 “It is my expectation that none of the guidelines prior to and including SIGN 114 (excluding SIGN 88, SIGN 93 and SIGN 101, which have been updated in the last three years) will have a record of the declarations of interest of the group members. This totals forty-four guidelines, of which six will have updates published in 2014 (with current declarations of interest)(3).”
We would urge SIGN and other guideline authors finding themselves in the same position to make it clear that current standards on transparency have not been met for a significant number of clinical guidelines still in use.
(1) Lenzer, J Evidence Based Medicine: Why we can’t trust clinical guidelines
Published 14 Jun 2013 BMJ 2013; 346 doi:
(2) SIGN Guidelines. Policy on Declaration of interests. File name: 20111219. Dated 23 Jul 2013 http://www.sign.ac.uk/pdf/doi-policy.pdf
(3) Programme Lead for SIGN Guidelines. E-mail communication to Dr Peter J. Gordon. 20 Nov 2013.
Competing interests: No competing interests
At the end of her article, Jeanne Lenzer mentions conflicted guidelines being a widespread problem. This view is supported by our analysis of guidelines in Germany. We evaluated the influence of drug companies on clinical trial findings and expert opinion with two examples (1).
Clinical trial findings which had been manipulated by the market authorization holder of gabapentin subsequently served as a basis for recommendations in German guidelines to prescribe gabapentin. To analyze the possible effects of financial ties between guideline authors and drug companies, a German guideline on the treatment of psoriasis vulgaris with efalizumab was compared with guidelines whose authors had no conflicts of interest. We found that Efalizumab was judged more favorably in the German guideline than in a guideline issued by the National Institute of Health and Care Excellence. For example, the evidence in its favor was judged as good, the use of efalizumab for induction and combination therapy in psoriasis vulgaris was recommended, and efalizumab was said to improve patients’ health-related quality of life.
Therefore, we are very supportive of the BMJ’s open data campaign. Public access to all trial data must be ensured so that independent evaluations are possible. We also think that the responsibility for creating guidelines should be borne by authors and organizations that do not have financial connections with pharmaceutical companies. International recommendations about the declaration and management of conflicts of interests should also be taken into account (2).
1. Schott G, Dunnweber C, Muhlbauer B, Niebling W, Pachl H, Ludwig WD: Does the pharmaceutical industry influence guidelines? Two examples from Germany. Dtsch Arztebl Int 2013; 110: 575-583.
2. Committee on Conflict of Interest in Medical Research, Education, and Practice, Institute of Medicine: Lo B, Field MJ (Hrsg.): Conflict of Interest in Medical Research, Education, and Practice. 1. Aufl., Washington D.C.: National Academies Press, 2009.
Competing interests: No competing interests
Dr. Young's letter adds little to the comments published by several authors in response to this pair of articles.
He wrangles over the details of calculations. With his proposed version, there have now been several estimates of the increased deaths due to MPSS treatment in acute SCI: about 3000 (Dr. Geisler), or <500 (Dr. Bracken), or about 200 (Dr. Young). As we have emphasized all along, many different calculations are possible, each depending on a particular set of assumptions, none of them uniquely without controversy.
The point is that he does not dispute that MPSS has caused a number of extra deaths. He does diligently find an optimistic set of assumptions to minimize the estimate. What he does not address is the original question why any number of deaths -- 200 or 500 or 3000 -- is an acceptable price to pay for a treatment whose benefits are unclear and have been questioned throughout an extensive literature.
More, Dr. Young does not answer Ms. Lenzer's main question. Why, instead of our usual reliance on strict evaluation by regulatory agencies like the FDA and on consensus in the open scientific community, do NASCIS and Dr. Bracken expect us to rely solely on their private judgment and authority? The NASCIS publications give positive coverage to efficacy and are explicit in their guidance to use MPSS; but we only discover now, afterwards, that there are implied, important caveats excluding the head and chest injuries and multiple trauma commonly found in actual SCI clinical practice.
Access to NASCIS data is only by a process that gives Dr. Bracken one-sided discretion over the uses to which his data can be put. This secrecy runs counter to the worldwide trend in the last few decades to require free availability of anonymized clinical trial data, especially in government-funded studies. Oversight has been ad hoc, not involving FDA scrutiny and approval. The NIH sent a fax to US emergency rooms suggesting MPSS treatment. The AANS/CNS guidelines committee has twice declined to endorse MPSS. ATLS has withdrawn its endorsement. The positive Cochrane Reviews article was written by Dr. Bracken himself.
Dr. Young suggests, "a randomized placebo-controlled clinical trial of MP for acute spinal cord injury would meet clinical equipoise and should help settle this issue." What he does not explain is how a human-volunteers committee could find such a study ethical. The available evidence suggests MPSS is not safe in acute SCI: Drs. Young and Bracken both admit that. Although the exact number may be "speculative," there are extra deaths, perhaps many. How could it be acceptable to expose experimental subjects to this apparent risk, when the benefits are poorly reported, widely regarded as unproven, and no more than "modest"--even in Dr. Bracken's later, more level estimation?
Competing interests: No competing interests
I’d like to thank Wise Young for his careful analysis of deaths due to high-dose steroids for acute spinal cord injury. He makes an important point: the number of attributable deaths from the treatment is not entirely clear, since all estimates are based on head and not spine injuries. This is a point acknowledged by Dr Geisler. While it is possible that Dr Geisler’s estimate is correct, I should not have used his comparison to the number of deaths in the World Trade Center attacks, since it evokes an emotional response that detracts from the primary issue: There appears to be little benefit and definite harm (whether it is the low estimate provided by Dr Young or the higher estimate provided by Dr Geisler), which led the American Association of Neurological Surgeons and the Congress of Neurological Surgeons to warn doctors not to use high-dose steroids for acute spinal cord injury because of their finding that “Class I, II, and III evidence exists that high-dose steroids are associated with harmful side effects including death.”1
I thank both Drs Young and Geisler for their comments.
1. Hurlbert RJ, Hadley MN, Walters BC, et al. Guidelines: Pharmacological Therapy for Acute Spinal Cord Injury. Neurosurgery. 2013;72:93-105. http://journals.lww.com/neurosurgery/toc/2013/03002.
Competing interests: author of article
Dr Messe, his co-authors and I agree on one thing: opinion is not science. However, Messe et al, mislead BMJ readers by suggesting I used the polls to prove whether a treatment is good or not. Quite the contrary, I cited the polls because it is incumbent on the guideline sponsors to explain why they failed to include even a single sceptic on the guideline writing panel when the majority of emergency physicians, including some prominent and respected experts, do not believe the science supports the guidelines. Science is supposed to be a rigorous process that includes attempts to disprove the theory in question. For that reason, I cited a recent unpublished poll of 548 emergency physicians in which only 16% said they support the new guidelines I included older published polls that show the new poll wasn’t a fluke. When there is this level of scientific debate, guideline panels should include sceptics.
Readers may be interested to note that Messe, Smith and Gronseth served on the tPA for acute stroke guideline panel, and each acknowledges ties to the manufacturers of tPA during the guidelines writing process. The fourth signatory is an administrator for one of the sponsoring organisations, which has received generous funding from the manufacturers.
I have to wonder if Messe et al, actually read my article, because their letter criticizing it evades the article’s central point: too often, industry influence has led to panel stacking in which panellists who are known to have published or expressed industry-friendly views are preferentially selected. Messe et al, not only fail to respond to that critical point, but they cite the endorsement of tPA by multiple guideline panels – most comprised of panellists who have ties to the manufacturer – as proof that it must be safe and effective. Using this line of reasoning, those of us concerned about climate change should stop worrying, since among white papers put out by panels supported by the oil industry, there is agreement that global warming is a myth.
If Messe et al, want a real examination of the science behind tPA for stroke, they should have included expert sceptics on the guideline panel instead of criticizing and attempting to debate the journalist who reported the problem. For readers who want to learn about analyses of tPA for stroke that are independent of industry and its researchers, I refer them to the citations that follow. 1-6
1. Fatovich DM. Tissue plasminogen activator for acute ischaemic stroke. Med J Aust. Apr 21 2008;188(8):489; author reply 490. http://www.ncbi.nlm.nih.gov/pubmed/18429725.
2. Hoffman JR, Cooper RJ. Stroke thrombolysis: we need new data, not more reviews. Lancet Neurol. Apr 2005;4(4):204-205. http://www.ncbi.nlm.nih.gov/pubmed/15778096.
3. Hoffman JR, Schriger DL. A graphic reanalysis of the NINDS Trial. Annals of emergency medicine. Sep 2009;54(3):329-336, 336 e321-335. http://www.ncbi.nlm.nih.gov/pubmed/19464756.
4. Newman D. The Guideline, The Science, and The Gap. Scientific Medicine and Research Translation, Emergency Medicine. 2013. http://www.smartem.org/content/guideline-science-and-gap.
5. Radecki RP. Skepticism about thrombolytics in stroke is not unreasonable. Nat Rev Neurol. 2011;8(3):176. http://www.ncbi.nlm.nih.gov/pubmed/22249840.
6. Radecki RP. Pharmaceutical sponsorship bias influences thrombolytic literature in acute ischemic stroke. West J Emerg Med. Nov 2011;12(4):435-441. http://www.ncbi.nlm.nih.gov/pubmed/22224134.
Competing interests: author of article
It is my understanding that AHA/ASA guidelines(authors included in some of the responders to this article) for rt-PA include selected patients in the 3-4.5 hour window. The European Stroke Organization has the same guidelines.
Jauch, EC, Saver, JL, Adams, HP, et al. Guidelines for the early management of patients with acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke. 2013 Mar;44(3):870-947
Competing interests: No competing interests
Jeanne Lenzer wrote in “Why we can’t trust clinical guidelines”  that “one expert estimated that more patients had been killed by the treatment in the past decade than died in the 9/11 World Trade Center attacks.” The treatment was methylprednisolone (MP) and the expert was neurosurgeon Fred Geisler who assumed that 10,000 people with spinal cord injury (SCI) received high-dose MP each year for 22 years, that half of them had multiple injuries that predisposed them to MP-induced complications, and that these people had a 3.4% increased risk of death extrapolated from the MRC CRASH trial  of 48-hour MP treatment of traumatic brain injury (TBI). Geisler estimated that 3740 spinal-injured people died from MP usage, which he points out is more than the 2996 people who died in the 9/11/01 terrorist attacks of New York and Washington DC. I write to question the claim of higher mortality due to MP treatment of SCI and the suggestion that MP has killed more people with SCI than the number of people who died in the 9/11/01 attack on New York for the following reasons.
• First, Geisler applied the overall 3.4% mortality increase from the CRASH study to acute SCI. As pointed out by Michael Bracken, mild TBI from the MRC CRASH study  is the category that is comparable with acute SCI. Severe TBI is associated with much higher mortality rates than SCI. In CRASH, the overall mortality rate in the control group was 22.3% (1075/4819) compared to 25.7% (1248/4854) in MP-treated group. The 6-month mortality rates of subjects with mild head injury in CRASH were 6.3% (89/1420) vs. 5.8% (86/1483) respectively for 48-hour MP-treated and control subjects, close to 5-7% mortality rates for placebo and MP-treated SCI observed in the NASCIS trials [3, 4]. If Geisler had used data from mild TBI from CRASH, the increased mortality risk would be a more reasonable 0.5% (i.e. 6.3%-5.8%) rather than 3.4% (i.e. 22.3%-25.7%).
• Second, Geisler assumed that 100% of 10,000 acute SCI patients per year for 22 years (1991-2013) received the MP treatment used in the CRASH trial. This is not possible. The CRASH trial used a 48-hour high-dose MP that NASCIS 3 recommended in 1997 only for patients treated between 3-8 hours after injury. The 24-hour course of MP is supposed to be given only if the therapy could be started within 3 hours. High-dose MP was not recommended for people with gunshot wounds or multiple traumas, both conditions excluded by NASCIS 2 and 3. High-dose MP was used in less than 60% of acute SCI patients. After 2007, use of high-dose MP fell to less than 40% of acute SCI in the U.S. Thus, probably less than 20% of people with acute SCI received the 48-hour course of MP between 1997 and 2013.
• Third, Lenzer stated that Geisler’s estimate is for the “the past decade” but Geisler said that his calculations were for 22 years. If only the past decade were considered, even using the erroneous assumptions that 10,000 people/year with SCI received the 48-hour course of MP and that 50% of these people were subject to the 3.4% mortality risk observed in CRASH, the number of people affected would be 1,700. Using more realistic assumptions that 20% of 10,000 people with new SCI received the 48-hour MP therapy each year between 1997 and 2013 and MP treatment killed 0.5% of them, this adds up to 160 people, a far cry from 3740 that Geisler estimated.
• Finally, there is no credible evidence that a 24-hour or 48-hour course of high-dose MP increases mortality in SCI . NASCIS 2  showed no significant difference amongst placebo-treated (12/171 or 7.0%), 24-hour MP-treated (7/166 or 4.2%), and naloxone-treated subjects (10/154 or 6.5%). NASCIS 3 likewise revealed no significant differences amongst 24-hour MP-treated patients (9/166, 5.4%), 48-hour tirilazad mesylate-treated patients (12/167, 7.2%), and 48-hour MP-treated patients (10/166, 6.0%). Several other studies [6-8] also found no significant differences of mortality rates between untreated and MP-treated patients in acute SCI. Chikuda, et al.  recently reviewed 3508 patients who had cervical SCI in Japan, 824 of whom received high-dose MP (>5000 mg). MP-treated patients had higher risks of gastrointestinal bleeding (8.6%) than unmatched controls (3.1%, p<0.001) or propensity matched controls (3.8%, p<0.001). However, in-hospital mortality rates of MP-treated patients (2.8%) did not differ significantly from mortality rates of unmatched controls (3.4%, p=0.485) or a propensity-matched control group (3.1%, p=0.884).
So, let us stop indulging in speculative estimates and focus on evidence instead. Given the controversy regarding the efficacy of MP and the dearth of alternative therapies for acute SCI, a randomized placebo-controlled clinical trial of MP for acute spinal cord injury would meet clinical equipoise and should help settle this issue.
1. Lenzer, J., Why we can't trust clinical guidelines. BMJ, 2013. 346: p. f3830.
2. Edwards, P., et al., Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroid in adults with head injury-outcomes at 6 months. Lancet, 2005. 365(9475): p. 1957-9.
3. Bracken, M.B., et al., Administration of methylprednisolone for 24 or 48 hours or tirilazad mesylate for 48 hours in the treatment of acute spinal cord injury. Results of the Third National Acute Spinal Cord Injury Randomized Controlled Trial. National Acute Spinal Cord Injury Study. Jama, 1997. 277(20): p. 1597-604.
4. Bracken, M.B., et al., A randomized, controlled trial of methylprednisolone or naloxone in the treatment of acute spinal-cord injury. Results of the Second National Acute Spinal Cord Injury Study. N Engl J Med, 1990. 322(20): p. 1405-11.
5. Bracken, M.B., Steroids for acute spinal cord injury. Cochrane database of systematic reviews, 2012. 1: p. CD001046.
6. Tsutsumi, S., et al., Effects of the Second National Acute Spinal Cord Injury Study of high-dose methylprednisolone therapy on acute cervical spinal cord injury-results in spinal injuries center. Spine, 2006. 31(26): p. 2992-6; discussion 2997.
7. Suberviola, B., et al., Early complications of high-dose methylprednisolone in acute spinal cord injury patients. Injury, 2008. 39(7): p. 748-52.
8. Ito, Y., et al., Does high dose methylprednisolone sodium succinate really improve neurological status in patient with acute cervical cord injury?: a prospective study about neurological recovery and early complications. Spine (Phila Pa 1976), 2009. 34(20): p. 2121-4.
9. Chikuda, H., et al., Mortality and morbidity after high-dose methylprednisolone treatment in patients with acute cervical spinal cord injury: a propensity-matched analysis using a nationwide administrative database. Emerg Med J, 2013. [10.1136/emermed-2012-202058]
Competing interests: Dr. Young was an investigator in both NASCIS 2 and 3 trials and an author of publications from these two trials. He has received travel support and honoraria from Upjohn, Pharmacia, and Pfizer to speak at conferences. He runs ChinaSCINet, a clinical trial network that tests therapies for SCI. He serves as a voluntary (unpaid) global medical director for Stemcyte, helping organize their research and clinical trials. Stemcyte donates umbilical cord blood to ChinaSCINet. Dr. Young helped found Acorda Therapeutics in 1995 and served on its Board of Directors from 1995-2012. He founded the Journal of Neurotrauma and served as its Editor in Chief from 1984 to 1991.
Instead of commenting on the merits or faults of Ms. Lenzer’s article, Dr. Radecki chose to rebut the Stroke neurologists’ replies to the article from an arbiter position. Well, Dr. Radecki, allow me to retort.
First, I give rt-PA to a minority of patients with stroke, only within the guidelines. The recommendations of these guidelines are supported by the very meta-analysis that you cite as proof against rt-PA. As is well known, no guideline proposes use of rt-PA beyond 3 hours.
Second, I do not lament the patients who did not receive rt-PA and were victims of what you call “conservative counter-reaction”. It is reasonable to have doubts and question any kind of evidence coming from any kind of source. I’ve been there. I’ve held rt-PA administration in patients when I was in doubt. However, as the evidence for the benefit of early rt-PA administration mounts, the passive aggressive opposition to its use by cherry-picking the evidence that you like may harm future patients.
Third, I do not denigrate the valid scientific enquiry of the skeptics. From the replies we read here, there is no such valid scientific enquiry, only opinions. It’s easy to say that the results are not perfect. We are far from a perfect solution. We shouldn't throw out the baby with the bath water.
Finally, I as well as all of the Stroke neurologists that you rebut here are against corruption of science by the pharmaceutical industry. However, we are equally opposed to the kind of yellow journalism that has sparkled this discussion.
Competing interests: Publication honorarium from MedLink.
The contribution by Dr. Messe, et al. to the continued discussion of Ms. Lenzer's article, unfortunately, adds little to the discussion of how professional or financial conflicts-of-interest were managed during writing the ACEP/AAN guidelines in question. Furthermore, Messe, et al. regrettably mislead through omission or imputation in their response:
• The authors cite Towfighi, et al. in the context of guidelines in support of tPA and use of tPA as a quality measure, stating "these endeavors have likely contributed to the improving outcomes for stroke patients". Towfighi, et al. do not mention tPA in the cited article, and attribute the decline in stroke mortality almost entirely to better stroke prevention.
• The authors selectively cite the favorable adjusted odds ratio for the IST-3 subgroup treated within 3 hours, but fail to mention the IST-3 authors specifically tested for an onset time-to-treatment (OTT) effect against the 3-4.5 hour subgroup (OR 0.73, 95% CI 0.50-1.07) and the 4.5-6 hour subgroup (OR 1.31, 95% CI 0.89-1.93) and found no observed association (p = 0.613). It is also a fallacy to cite a single post-hoc subgroup statistical outcome as refutation of the overall negative primary outcome of the largest stroke trial in history.
• The authors cite positive findings from the meta-analysis from Lees, et al. It should again be noted the majority of these studies were small, manufacturer-sponsored, and failed to demonstrate benefit exceeding harms – including a trial that was modified due to harms, and then stopped early for futility. The risk of bias in this data set is high, and its results ought to be viewed as informing future trials, rather than reliable proof.
• The authors cite the Cochrane review findings suggesting benefit up until six hours after onset. This review predates IST-3, and it should be noted the updated meta-analysis of alteplase inclusive of IST-3 regresses towards the mean and no longer suggests a treatment benefit in the 3-6 hour subgroup (OR 1.07, 95% CI 0.96-1.20).
Patient interests are not served by the selective citation of results on either side of the debate, nor are they served by segregation of points of view during the formulation of guidelines. The use of tPA in acute stroke stands as promising therapy without overstatement of the strength of the evidence; to do so impedes the best-practice dissemination Messe, et al. decry, as well as further research into maximizing benefit and minimizing risk.
1. Towfighi A, Saver JL. Stroke declines from third to fourth leading cause of death in the United States: historical perspective and challenges ahead. Stroke. 2011 Aug;42(8):2351-5.
2. IST-3 collaborative group. The benefits and harms of intravenous thrombolysis with recombinant tissue plasminogen activator within 6 h of acute ischaemic stroke (the third international stroke trial [IST-3]): a randomised controlled trial. Lancet. 2012 Jun 23;379(9834):2352-63.
3. Lees KR, Bluhmki E, von Kummer R, Brott TG, Toni D, Grotta JC, et al. Time to treatment with intravenous alteplase and outcome in stroke: an updated pooled analysis of ECASS, ATLANTIS, NINDS, and EPITHET trials. Lancet. 2010 May 15;375(9727):1695-703. doi: 10.1016/S0140-6736(10)60491-6.
4. Wardlaw JM, Murray V, Berge E, Del Zoppo GJ. Thrombolysis for acute ischaemic stroke. Cochrane Database Syst Rev 2009(4):CD000213.
5. Wardlaw JM, Murray V, Berge E, del Zoppo G, Sandercock P, Lindley RL, et al. Recombinant tissue plasminogen activator for acute ischaemic stroke: an updated systematic review and meta-analysis. The Lancet 2012;379(9834):2364-72.
Competing interests: No competing interests