BMJ, doi: 10.1136/bmj.39504.662685.0F, (Published 27 February 2008)

Feature

Antidepressants

An untold story?

A recently published study analysing the effectiveness of some antidepressants highlights the ongoing problem of how study results may be distorted by failure to make data fully available. Jeanne Lenzer and Shannon Brownlee report

New generation antidepressants aren’t all they’re cracked up to be. That seems to be the central message in the meta-analysis published this week by Irving Kirsch and colleagues in PLoS Medicine,¹ and it was this message that made the headlines. Kirsch’s conclusion follows on the heels of similar studies showing that statins are useful in only a small subset of patients taking the drugs² and earlier studies finding that the safety and performance of cyclo-oxygenase-2 (COX 2) inhibitors were worse than they first seemed.³ All of which further reinforces previous criticisms that regulators in the United Kingdom and the United States are not doing their duty to protect the public from useless or dangerous drugs. But there’s another, deeper problem here—a problem that, ironically enough, was highlighted by GlaxoSmithKline’s news release stating that Kirsch’s conclusions are "incorrect" because he evaluated only a "small subset of the total data available." How can regulators, the public, and doctors know how useful (or how potentially dangerous) drugs really are unless outside researchers have access to all the data?

The gist of Kirsch’s analysis of published and unpublished data from studies of antidepressants in adults is that only a very small subset of patients seemed to benefit to a clinically significant degree. But even now, do we really know the truth about antidepressants? Or statins? Or any other drug on the market? Even when researchers make exhaustive searches and file requests with regulatory authorities under freedom of information legislation, the datasets they need to validate results can remain tantalisingly out of reach.

Ending publication bias: necessary but not enough

Lack of access to data is an ongoing problem in the United States, despite passage of the Food and Drug Administration Amendments Act (FDAAA) of 2007, which was intended to make clinical trial data more transparent. The act requires most clinical trials to be registered and their results to be posted at ClinicalTrials.gov, a clinical trials registry of the National Library of Medicine. It’s an admirable first step,⁴ but the FDAAA may not reduce the likelihood of dangerous or ineffective drugs remaining on the market as much as some people might have hoped. For one thing, it fails to fully overcome the problem of publication bias, in part because some studies do not have to be registered or have their results posted (such as preclinical or toxicity trials or trials for drugs and devices that fail to win approval for any indication). But access to full data is also constrained by other policies and laws, such as trade secrecy laws, that prohibit the release of some important clinical data—data not carried by ClinicalTrials.gov.

A 2008 study by Erick Turner and colleagues in the New England Journal of Medicine highlights the problem of publication bias.⁵ They found that 8% of antidepressant trials with negative findings were reported as negative, while positive trials were reported as such 97% of the time. Publication bias is not limited to antidepressants. A former clinical reviewer for the FDA, Turner told the BMJ that it is critical for researchers to be able to obtain complete study protocols and full datasets to be able to determine whether a study’s conclusions are valid. His concerns were highlighted by a 1999 study that looked at papers published in five top medical journals.⁶ It found that in 18% to 68% of articles (depending on journal) the authors’ conclusions as stated in journal abstracts either were not supported or were contradicted by data given in the body of the article.

To overcome publication bias, researchers often request data from the FDA. That’s what Kirsch and his colleagues did. They filed a request under the Freedom of Information Act with the agency for all study data for the six most widely prescribed of the new generation antidepressants on the market. The FDA identified 47 relevant trials, but for reasons known only to itself the agency failed to release data from nine of the trials. The FDA told the BMJ that it would look into the reasons for the failure to release the nine studies—which, as it happens, all yielded negative results. Although Kirsch was able to get some data from drug makers, he was ultimately unable to obtain data for 38% of test participants in trials of sertraline and 23% in trials of citalopram—making it impossible to analyse two of the six antidepressants for overall efficacy.

Part of the problem here, of course, is that to be able to ask for study data researchers must know that a study has been done. Until now this information was often not easy to come by. Companies could legally refuse to reveal that they were even conducting certain studies of drugs already on the market. The FDAAA remedies this problem by requiring researchers and companies to post the primary and secondary outcome measures of their studies at the time of registration (generally within 21 days of enrolment of the first patient) and the results within one year (with extensions up to two years) of the time that the FDA approves the drug or some other action is taken or the trial is concluded.

This is all well and good. But the law does not require the registration of studies that were performed in the past for drugs that are currently on the market. Should researchers wish to perform studies like that of Kirsch and his colleagues, ClinicalTrials.gov won’t make it much easier for them. Nor does the FDAAA mandate the release of underlying data. Instead, only the key summary results must be posted. Deborah Zarin, director of ClinicalTrials.gov, says that the registry will require summary results for primary and secondary outcome measures in tabular form, though details are still being worked out.

In some circumstances, that may be enough. But there are a number of cases in which it isn’t. Without access to underlying data being fully available to researchers, patients are at risk of serious harm. Fred Geisler, a neurosurgeon at the Illinois Neuro-Spine Center, points to the use of high dose steroids in patients with spinal injury on the basis of a single, potentially flawed study funded by the National Institutes of Health (NIH).⁷ Geisler believes that several thousand patients have died as the result of high dose steroids used to treat acute spinal cord injury. Two recent surveys show that most neurosurgeons share his concerns.⁸ They think that steroids are either useless or dangerous; yet when asked why most of them continue to give the drug, they cite fears of malpractice on the basis of the standard of practice set by the NIH study. Several researchers have lobbied unsuccessfully for the release of the underlying data, without which they cannot verify their concerns—or lay them to rest.

When should a clinical outcome be a trade secret?

Perhaps the most daunting obstacle to full access to data—and one that the FDAAA doesn’t deal with—is the trade secrecy rules that allow the FDA and industry to prevent disclosure of critical data, regardless of what they are now required to post on ClinicalTrials.gov. This aspect of drug regulation surfaced in 2005, with the death of 19 year old Traci Johnson, who committed suicide while serving as a healthy volunteer in a trial of duloxetine for a new indication, urinary incontinence. After requesting the data on duloxetine from the FDA, one of us (Lenzer) found that Johnson’s death, in addition to those of at least four other volunteers, was not included. When questioned, the FDA cited trade secrecy laws,⁹ which permit companies to withhold all information, even deaths, about drugs that do not win approval for a new indication, even when the drug is already on the market for other indications.

The potential risk to patients should be obvious. Take valdecoxib, a COX 2 inhibitor that was withdrawn from the market because it posed a serious risk of heart attacks.¹⁰ In 2004 the health research group of Public Citizen, a non-profit, public interest organisation in Washington, DC, tried to assess the drug’s safety profile. In 2001 the manufacturer applied for approval to market valdecoxib for four indications: osteoarthritis, dysmenorrhoea, adult rheumatoid arthritis, and acute pain. The FDA approved the drug for the first three indications but not for acute pain, and some of the information about the acute pain trials was withdrawn from the FDA website and a statement given that the information contained "trade secret and/or confidential information that is not disclosable."¹¹ That left the researchers and the public in the dark. Did the drug fail to relieve acute pain? Did it have serious side effects? It is precisely these failed trials that should be made public.

Similarly, when Peter Jüni and colleagues requested data from the FDA on valdecoxib, they too received a page of material so heavily censored that it looked more like a military intelligence document than a medical study.³ Redaction, the censorship of certain information in a document that is released, is carried out under exemption 4 of the Freedom of Information Act, which allows trade secrets or "protected commercial information" to be concealed. But what commercial interest is there in censoring the clinical outcomes of participants in clinical trials? Should a death be considered a trade secret? Should lack of efficacy be a trade secret? Sidney Wolfe, director of Public Citizen’s health research group, said, "I’ve never been able to get any kind of protocol for what [FDA staff] are instructed to redact, but in general they redact way more than they should." He added, "Of course, it’s a catch 22, because if you don’t know what they are redacting you can’t argue that it should not have been redacted."

Alastair Wood, the head of the FDA’s advisory committee on the safety of COX 2 inhibitors, insists that there is no reason ever to redact clinical trial data. "There are some things like manufacturing data that might be commercially sensitive information," he said, "but that’s not in clinical trials." So, how can withholding statistical reviews and clinical data be justified? Without knowing what information is being withheld or the rules guiding redaction, the interpretation of what constitutes a trade secret seems itself to be a trade secret.

The universe of data

Turner, author of the study on publication bias and antidepressants, says that although the ClinicalTrials.gov database is a positive step towards greater transparency he believes that an excellent database already exists. The FDA database, he says, should be made available to researchers. Unlike the ClinicalTrials.gov database, which includes information only from studies that either started or had not concluded by the end of 2007, the FDA database can reach back to provide data on the overwhelming number of drugs already on the market. Furthermore, the FDA database contains a vast array of scientific reviews, while the results posted at ClinicalTrials.gov will be far more rudimentary. The FDA says that it is far too onerous to put all its material online, especially as it would have to comb through all the data for trade secrets. But, as a number of experts have pointed out, the burden on the FDA from future requests made under the Freedom of Information Act would be lessened if it posted all its data.

Beyond studies of drugs and devices already on the market—or that will come onto the market—there are data from failed trials of drugs that never win. These data should also be preserved for researchers. Currently, when a drug isn’t approved, all information about it is protected as a trade secret. At first blush it might seem unnecessary to learn about drugs that aren’t on the market at all. But, Wood says, "Suppose someone develops a drug for disease X and it either causes toxicity or doesn’t work. That’s something people should know."

But data from failed drug applications are protected as trade secrets so that drug companies aren’t put at a "competitive disadvantage" when other companies, learning of the initial studies, aren’t forced to expend the same "wasted efforts." Reproducing wasted efforts can mean wasted lives—and wasted money. It also violates the covenant between human participants and researchers by allowing them to be exposed to unnecessary risks. This is a question of ethics. Should a second company be allowed to launch a trial of a similar drug without the benefit of knowing it may pose a danger to participants? There should be a centralised database for such preclinical and failed trails, Wood says. "We shouldn’t be walking down paths that lead to nowhere."

Ultimately, forcing companies to expend "wasted efforts" so as to protect commercial interests, redacting clinical information from studies of drugs and medical devices, and failing to insist that data derived from the sacrifices of human volunteers be placed in the public domain simply can’t be reconciled with what is in the public interest. Trial participants, as well as patients who take drugs and doctors who prescribe them, deserve nothing less than the assurance that all the news—not just the good news—has been carefully assessed.

References

1. Kirsch I, Deacon BJ, Huedo-Medina TB, Scoboria A, Moore TJ, Johnson BT. Initial severity and antidepressant benefits: a meta-analysis of data submitted to the Food and Drug Administration. PLoS Med 2008;5(2):e45 doi: 10.1371/journal.pmed.0050045.[CrossRef][Medline]
2. Abramson J, Wright JM. Are lipid-lowering guidelines evidence-based? Lancet 2007;369:168-9.[CrossRef][Web of Science][Medline]
3. Jüni P, Reichenbach S, Egger M. COX 2 inhibitors, traditional NSAIDs, and the heart. BMJ 2005;330:1342-3.[Free Full Text]
4. Groves T. Mandatory disclosure of trial results for drugs and devices. BMJ 2008;336:170.[Free Full Text]
5. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252-60.[Abstract/Free Full Text]
6. Pitkin RM, Branagan MA, Burmeister LF. Accuracy of data in abstracts of published research articles. JAMA 1999;281:1110-1.[Abstract/Free Full Text]
7. Bracken MB, Shepard MJ, Holford TR, Leo-Summers L, Aldrich EF, Fazl M, et al. Administration of methylprednisolone for 24 or 48 hours or tirilazad mesylate for 48 hours in the treatment of acute spinal cord injury. Results of the third national acute spinal cord injury randomized controlled trial. National acute spinal cord injury study. JAMA 1997;277:1597-1604.[Abstract/Free Full Text]
8. Lenzer J. NIH secrets: study break. New Republic 2006 Oct 30. www.ahrp.org/cms/index2.php?option=com_content&do_pdf=1&id=398.
9. Lenzer J. Drug secrets: what the FDA isn’t telling. Slate 2005 Sep 27. www.slate.com/id/2126918/.
10. Waknine Y. Bextra withdrawn from market. Medscape Today 2005 Apr 7. www.medscape.com/viewarticle/502642?rss.
11. Public Citizen Health Research Group versus FDA—complaint (26 Feb 2004). www.citizen.org/documents/ACF5CF.pdf.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

The researcher’s credo

David O Antonuccio and David Healy
BMJ 2008 336: 629. [Extract] [Full Text] [PDF]

Excessively closed science hurts

Fred H. Geisler
BMJ 2008 336: 629. [Extract] [Full Text] [PDF]

Finding meanings

Jane Smith
BMJ 2008 336: 0. [Extract] [Full Text]

Bitter sweets

Des Spence
BMJ 2008 336: 562. [Extract] [Full Text] [PDF]

Meta-analysis shows difference between antidepressants and placebo is only significant in severe depression

Susan Mayor
BMJ 2008 336: 466. [Extract] [Full Text] [PDF]

Mandatory disclosure of trial results for drugs and devices

Trish Groves
BMJ 2008 336: 170. [Extract] [Full Text] [PDF]

COX 2 inhibitors, traditional NSAIDs, and the heart

Peter Jüni, Stephan Reichenbach, and Matthias Egger
BMJ 2005 330: 1342-1343. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

• Antonuccio, D. O, Healy, D. (2008). The researcher's credo. BMJ 336: 629-629 [Full text]  
• Geisler, F. H. (2008). Excessively closed science hurts. BMJ 336: 629-629 [Full text]  

Rapid Responses:

Read all Rapid Responses

Generic or Branded

Matthew J Jackson
bmj.com, 28 Feb 2008 [Full text]

“Anti-depressant’ Meta-Analysis Misleading to Public and of Limited Value to Clinicians

Fiona Moir, et al.
bmj.com, 28 Feb 2008 [Full text]

Differences between antidepressants and placebo

Francis J Dunne
bmj.com, 1 Mar 2008 [Full text]

Telling the antidepressant story

D B Double
bmj.com, 1 Mar 2008 [Full text]

What is clinical significance?

Richard K Morriss
bmj.com, 2 Mar 2008 [Full text]

Storm in an antidepressant cup

Riccardo N Caniato
bmj.com, 11 Dec 2008 [Full text]

Overprescribing of medications

Ruchi Thakur
bmj.com, 6 Mar 2008 [Full text]

NASCIS publications abused the processes of peer-review, publication, regulatory approval, and clinical consensus

Fred H. Geisler
bmj.com, 12 Mar 2008 [Full text]

How the NASCIS studies gamed the peer-review and regulatory processes

William P. Coleman
bmj.com, 7 Mar 2008 [Full text]

Universal Human Subjects Commitment

David O. Antonuccio, Ph.D., et al.
bmj.com, 7 Mar 2008 [Full text]

Demand, don't just ask

P Ken Gillman
bmj.com, 19 Mar 2008 [Full text]

Rumsfeld misquoted

Doug Altman
bmj.com, 25 Mar 2008 [Full text]

Methyprednisolone and acute spinal cord injury: redux

Michael B Bracken
bmj.com, 14 May 2008 [Full text]

Corticosteroids could harm patients with spinal cord injury

Ian Roberts
bmj.com, 28 May 2008 [Full text]

Reply to Dr. Bracken

Fred H. Geisler, et al.
bmj.com, 10 Jun 2008 [Full text]

Online poll

Find out more>>