Rapid Responses to:
|
|
Rapid Responses: Rapid Responses published:
-
![[Read Rapid Response]](/icons/misc/sectionDOWN.gif)
Time to slow down and accentuate the negative
- Trevor Lane, DPhil (28 July 2006)
-
Beware abstracts
- Tom Hughes-Davies (30 July 2006)
-
Transparency is a must
- Jeevan P Marasinghe, Amarasinghe A A W,MD, Mcdonough, Georgia, USA. (31 July 2006)
-
P-values without value?
- James Penston (4 August 2006)
-
A statistical quibble
- Frank E Harrell (23 August 2006)
|
|
|||
|
Trevor Lane, DPhil, Technical writer Hong Kong
Send response to journal:
|
Gotzsche (1) has found that significant results were overreported in abstracts in 2003 and casts doubt on their credibility. It is widely accepted that publication bias has led to the selective reporting of positive results and underreporting of negative ones. Compulsory trial registration and publication may help restore the balance in terms of publishing negative trials. However, the constant pressure to save time and resources may also be a factor in the overreporting and lack of reliability of positive results. Many will admit to reading only abstracts to obtain information quickly, and BMJ editors have since before 2004 done so during initial manuscript screening to reject substandard original articles (2). Although Gotzsche does not name any journals, and it is likely that other journals screen abstracts to save time and money, this practice could have inadvertently contributed to biases in statistical reporting in abstracts to raise chances of publication. Of course, authors who rush to publish may also make errors. In addition, 11% (48/435) of supposed randomised controlled trials retrieved by Gotzsche actually were not randomised controlled trials, and the power of most was low. Surely, adherence to CONSORT guidelines (3) and efficient peer review should have prevented these errors. Authoritative journals need to increase their reliance on (or quality of) statisticians, not only to review methodology according to checklists (4), but also to recalculate data from all abstracts, and ideally to review all raw data, as Gotzsche hints. Authors could be charged at-cost, and these charges could be routinely included in grant submissions. Gotzsche does not say whether unstructured or structured abstracts were more reliable; major medical journals use the latter despite their typical underreporting of shortcomings (5). Perhaps structured abstracts need to include a statement of the calculated sample size, as well as a Limitations and Recommendations section to force authors to reveal any negative aspects of their studies. Wrong results and interpretations in abstracts together with wrong information in the main paper are a cause for concern if they are carried over to policymakers, media, public, and patients. We need to slow down and enforce checks for the sake of accuracy, reliability, and accountability, as well as encourage publication of negative studies. 1. Gotzsche PC. Believability of relative risks and odds ratios in abstracts: cross sectional study. BMJ 2006;333:231-4. 2. http://bmj.bmjjournals.com/cgi/content/full/329/7464/470 3. http://www.consort-statement.org 4. http://bmj.bmjjournals.com/advice/checklists.shtml#peer 5. Wong HL, Truong D, Mahamed A, Davidian C, Rana Z, Einarson TR. Quality of structured abstracts of original research articles in the British Medical Journal, the Canadian Medical Association Journal and the Journal of the American Medical Association: a 10-year follow-up study. Curr Med Res Opin 2005;21:467-73. Competing interests: None declared |
|||
|
|
|||
|
Tom Hughes-Davies, Paediatrician SP6 2EJ
Send response to journal:
|
Abstracts are often unintentionally misleading to the inexpert. Perhaps they should be written with the peer reviewers (who in a general journal should include a generalist) rather than by the authors alone. Competing interests: None declared |
|||
|
|
|||
|
Jeevan P Marasinghe, Registrar in Obstetrics and Gynecology,General Hospital( Teaching), Peradeniya, Sri Lanka. 20300, Amarasinghe A A W,MD, Mcdonough, Georgia, USA.
Send response to journal:
|
The scope of medical literature is expanding day by day and findings of medical research are often met with considerable scepticism. Some may question the quality and applicability of published material for day to day practice and this has eroded the public and academic confidence on publications. This may terminate in poor justification of the value and the importance of evidence based medicine (1), (2), (3).So it is essential to overcome this growing problem to safe guard the future clinicians and the health of the society. Gotzche (4) found that most results in abstracts were statistically significant. He has come across only five trials which had p value of 0.05<p<0.06, where as 29 trials had 0.04<p<0.05.This could well be due to incomplete reporting of outcomes within published articles (5), (6) and attributable to the human nature of the investigators and the sponsors of trials who show a strong tendency to accumulate positive outcomes than null hypothesis. (R .A Fisher, the founder of statistical inference, stated that p<0.05 was a standard level for concluding that there was evidence against null hypothesis tested.)But the danger is that deliberate publishing of positive results and unpublished negative results may lead to erroneous interpretation. Application of such study results may finally give rise to inappropriate management plans, protocols or guidelines of patient management. This depicts the importance of transparency and full disclosure of clinical research (7).Genuine approach with a paradigm shift to clinical research can improve transparency leading to patient’s safety and suppress the emerging disbelief on significant results in abstracts. Referrrences. (1).Kliche T.The development of evidence based prevention. Health promotion and education as an approach to continuous quality improvement of health care.Bundesquesundheitsblatt Gesundheistforschung Guesundheitsschutz.2006 Feb; 49(2):141-50. (2)Rosenstock L, Lee LJ.Attacks on science: the risk to evidence based policy. Am J Public Health.2002 Jan; 92(1):14-8. (3)Palmer GR.Evidence based health policy making, hospital funding and health insurance. Med J Aust.2000 Feb 7; 172(3):130-3. (4)Peter C Gotzsche.Believability of relative risks and odds ratios in abstracts: cross sectional study.BMJ 2006; 333:231-234(29 July). (5)An-Wen Chan, Douglas G Altman. Identifying outcome reporting bias in randomized trials on PubMed: review of publications and survey of authors.BMJ 2005; 330:753(2 April) (6)Karmela Krleza-Jeric et al Principals for international registration of protocol information and results from human trials of health related interventions: Ottawa statement.BMJ 2005; 330:956-958(23 April). (7).Altman DG, Schulz KF, Maher D, et al for the CONSORT group .The revised CONSORT statement for reporting randomized control trials: explanation and collaboration. Ann Intern Med 2001; 134:663-94. Competing interests: None declared |
|||
|
|
|||
|
James Penston, Consultant Physician/Gastroenterologist Scunthorpe General Hospital, Scunthorpe, North Lincolnshire DN15 7BH
Send response to journal:
|
Sir, Peter Gotzsche [1] draws attention to the skewed distribution of P- values in abstracts from a large sample of randomised trials and observational studies. Five studies reported P-values in the interval 0.05<P<0.6 compared with 46 in the interval 0.04<P<0.5. By itself, this finding may simply reflect reporting bias although, as he suggests, it leaves open the possibility of bias during the analysis. Given the pre-eminence of randomised controlled trials in medical research, the findings in respect of twenty-nine clinical trials reporting P-values in the interval 0.04<P<0.05 are of particular concern. Where sufficient data were available to allow the results to be checked, the P-values were confirmed in only eight of 23 studies. Why are so many of the results that creep into statistical significance unreliable? Is it that researchers are incompetent? Or are the incorrect results the product of research misconduct? Gotzsche is surely justified in urging caution when interpreting P- values in clinical research. His findings, though, may be just the tip of an iceberg. Whether due to incompetence or deliberate manipulation of data, there is no reason to believe that the reporting of false P-values – or, for that matter, any other statistical result – is confined to marginally significant results. The problem is that data from clinical trials are so complex, opaque and impenetrable that flaws are readily hidden. Peter Gotzsche deserves credit for exposing the unreliability of results in clinical research. References [1] Gotzsche P. Believability of relative risks and odds ratios in abstracts: cross sectional study. BMJ 2006;333;231-4. Competing interests: None declared |
|||
|
|
|||
|
Frank E Harrell, Chair, Department of Biostatistics Vanderbilty University School of Medicine
Send response to journal:
|
Dr Gotzsche has written an excellent article. I just wish to point out one minor statistical flaw. Some abstracts seemed to be criticized for using the chi-square test without continuity correction. The continuity correction is an attempt to make the Pearson chi-square test behave like Fisher's exact test. As Fisher's test is conservative (i.e., P-values from it are too large) this is not a good goal, and the uncorrected chi-square statistic actually provides more accurate P-values in general. See for example Agresti A, Stat in Med 20:2709; 2001. Competing interests: None declared |
|||