Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses
BMJ 2012; 344 doi: https://doi.org/10.1136/bmj.d7202 (Published 03 January 2012) Cite this as: BMJ 2012;344:d7202All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Thank you for these comments. We did not have the opportunity to evaluate the protocols for the trials included in our analysis. We focused our analysis on the results that were reviewed by the FDA, but not made available in scientific publications. As Dr. Shah notes, there are a number of reasons why a study may not be published. Previous research suggests that the most likely reason is that it is not submitted (1,2). The focus of our study was industry-sponsored trials that had been submitted to the FDA. Therefore, we did not have sufficient variability in funding source to conduct a sensitivity analysis.
1. Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results. Followup of applications submitted to two institutional review boards. JAMA 1992; 267: 374-378.
2. Lee, K, Boyd, E, Holroyd-Leduc, J, Bacchetti, P, and Bero, L. Predictors of publication: Characteristics of submitted manuscripts associated with acceptance at major biomedical journals, Medical Journal of Australia, 2006; 184: 621-626.
Competing interests: No competing interests
In this interesting paper we learn about the biases introduced by the use of incomplete data sets on drug meta-analyses; however, it is unfortunately not possible to reproduce the individual medications findings because of the limited information that is given in the methods section, associated with a lack of references.
For instance, table 2 refers to the finding of unpublished pre-2007 data about almotriptan migraine drug studies which would add an 11% of patients to the published data from a 2007 (following Table 1) meta-analysis. The 2007 meta-analysis reference is not given, although it might be the Chen et al. Headache 2007;47:1169-1177 paper (tbc).
The list of clinical trial/s (and their sources) where these 11% new almotriptan patients "unpublished data" would come from is also not described.
Although in this case the differences in the efficacy parameters results are small, I consider that it would be of great interest to communicate the references of the published meta-analyses you have used, as well as to know about the trial/s detail/s and sources from your "unpublished data", to allow anyone to reproduce your figures and find out more about which kind of data were missing.
Adding references of the reviewed meta-analyses (Table 1) and a new (large) table describing the sources of the unpublished data for each drug would solve the issue.
Competing interests: Employee of Almirall S.A., manufacturer of almotriptan, migraine medication mentioned in the paper
The article by Hart et al (1) found that psychiatric drugs, far more than drugs in other fields, were prone to reporting bias. Data predicting a lack of efficacy for anti-psychotics such as aripiprazole and ziprasidone have simply been suppressed. A similar attempt to ignore negative findings as to efficacy also is the case with anti- depressants.
The anti- psychiatrist author, Robert Whitaker, (2) has argued for years that the clinical trials for anti- psychotics are warped. He seems now a true prophet,however much he is vilified by the psychiatric establishment.
The avarice and greed of drug corporations have usurped the proper clinical assessment of anti- psychotics. There is an element of sulphur and brimstone about these drugs. They are not miracle cures for schizophrenia, but dangerous chemicals with ghastly side effects- from movement disorders to diabetes. Marketing and mammon do not mix well with medicine.
REFERENCES:
(1) Effect of reporting bias on meta- analyses of drug trials:reanalysis of meta-analyses. Beth Hart et al. BMJ 2012;344:d7202
(2) Mad In America:Bad Science, Bad medicine and the Enduring Mistreatment of the Mentally Ill. Robert Whitaker. 2001.
Competing interests: No competing interests
Dear Sir,
At the outset, I would like to congratulate the authors for the wonderful study. During such kind of analysis, following points may be useful:
1.Is the unpublished study for the pre decided objectives as per protocol? This can be verified by checking through clinical trial registry. As we know the value of results for post-hoc findings are many times questionable and may only generate hypothesis for future studies.
2.What are the probable reasons for not being published? This can be looked in two fashions: a) the opinion of the authors and b) opinion of the two authors doing this study. This may be required as published articles have mechanism of peer-review system.
3.Is any conflict of interest for non-publication from sponsoring agency- may be pharma industry or any other agency.
All these issues may also be included as part of sensitivity analysis for better understanding of very important issue which authors have tried to study in excellent way.
Dr.Pankaj B.Shah
Professor, Dept. of Community Medicine.
SRMC & RI, Chennai, India
Competing interests: No competing interests
Re: Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses
Thank you for your interest in our paper. We are happy to provide additional information that was not included in our paper. As you surmise, we did re-analyze the Chen article. We include the complete list of papers containing meta-analyses that we re-analyzed below.
All of the unpublished data that we included in our re-analyses came from publicly available Food and Drug Administration medical and statistical officer reports. The source of these data is described in detail in Rising, 2008 (1).
For almotriptan, in our 2008 paper (1), Trial CL11 was classified according to our pre-specified criteria as unpublished because it was published only as a conference proceeding (2). As noted in our previous publication, conference proceedings were classified as unpublished because they contained insufficient information to critically appraise the study and insufficient data for inclusion into a meta-analysis. However, it would be possible to contact the authors to obtain additional information.
Almotriptan was a drug for which the unpublished data had already been obtained by the meta-analysis authors and included in the published meta-analyses. As we note on page 4, second column, paragraph 4 of the article, for 10 of the 42 meta-analyses the unpublished trial outcome data had been identified and included. Therefore, our analysis involved removal of the Trial CL11 data from the meta-analysis.
1. Rising K, Bacchetti P, Bero L. Reporting bias in drug trials submitted to the Food and Drug Administration: A review of publication and presentation. PLoS Medicine 2008;5(11)e217.
2. Cabarrocas X. Efficacy and tolerability of subcutaneous almotriptan for the treatment of acute migraine: A randomized, double-blind, parallel-group, dose-finding study. Clinical Therapeutics 2001;23:11.
References for published meta-analyses that were reanalyzed:
Almotriptan:
Chen LC, Ashcroft DM. Meta-analysis examining the efficacy and safety of almotriptan in the acute treatment of migraine. Headache 2007;47:1169-1177.
Olmesartan:
Heran BS, Wong MMY, Heran IK, Wright JM. Blood pressure lowering efficacy of angiotensin receptor blockers for primary hypertension (Review). Cochrane Database Syst Rev 2009;4: CD003822.DOI: 10.1002/14651858.CD003822.pub2.
Reminyl:
Loy C, Schneider L. Galantamine for Alzheimer’s disease and mild cognitive impairment (Review). Cochrane Database Syst Rev 2009;1: CD001747. DOI: 10.1002/14651858. CD001747.pub3.
Cefditoren:
Casey JR, Pichichero ME. Meta-analysis of cephalosporins versus penicillin for treatment of group A streptococcal tonsillopharyngitis in adults. Clinical Infectious Diseases 2004;38:1535-7.
Frovatriptan:
Poolsup N, Leelasangaluk V, Jittangtrong J, Rithlamlert C, Ratanapantamanee N, Khanthong M. Efficacy and tolerability of frovatriptan in acute migraine treatment: systematic review of randomized controlled trials. Journal of Clinical Pharmacy and Therapeutics 2005;30:521-532.
Aripiprazole:
Bhattacharjee J, El-Sayeh HGG. Aripiprazole versus typical antipsychotic drugs for schizophrenia (Review). Cochrane Database Syst Rev 2010;1: CD006617. DOI: 10.1002/14651858.CD006617.pub3.
Pimecrolimus:
Ashcroft DM, Chen LC, Garside R, Stein K, Williams HC. Topical pimecrolimus for eczema (Review). Cochrane Database Syst Rev 2007;4:CD005500.
Ziprasidone:
Bagnall AM, Jones L, Ginnelly L, Lewis R, Glanville J, Gilbody S, Davies L, Torgerson D, Kleijnen J. A systematic review of atypical antipsychotic drugs in schizophrenia. Health Technology Assess 2003;7(13).
Eletriptan:
Pascual J, Mateos V, Roig C, Sanchez-del-Rio M, Jimenez D. Marketed oral triptans in the acute treatment of migraine: a systematic review on efficacy and tolerability. Headache 2007;47:1152-1168.
Competing interests: No competing interests