Rapid Responses to:
|
|
Rapid Responses: Rapid Responses published:
-
![[Read Rapid Response]](/icons/misc/sectionDOWN.gif)
Missing sample size calculations for "negative" trials
- Michael Power (13 May 2009)
-
Reporting sample size calculation is not more important for trials with statistically unsignificant results
- Pierre Charles, Agnès Dechartres (20 May 2009)
-
Reporting of sample size calculation
- Paul AVILLACH, L Rachid Salmi (21 May 2009)
-
Sample size calculations
- Jonathan D. Mayer (27 May 2009)
|
|
|||
|
Michael Power, Guideline developer Sowerby Centre for Health Informatics at Newcastle Ltd, NE1 2ES, UK
Send response to journal:
|
This study provides interesting data, but leaves unanswered the question: How often did trials finding no significant difference fail to publish a (correct) sample size estimate? When planning a trial it is obviously important to make an accurate estimate of sample size. However, when assessing published trials, sample size calculations are of real interest only for trials with non- significant differences in outcomes of compared interventions. Please could the authors provide this information. Competing interests: None declared |
|||
|
|
|||
|
Pierre Charles, research fellow in epidemiology, specialist registrar in internal medicine Hôpital Bichat Claude Bernard, 46 rue Henri Huchard,75877 Paris Cedex 18, France, Agnès Dechartres
Send response to journal:
|
In our survey, 57 articles (55%) of the 103 articles with unsignificant results reported all the required parameters for the sample size calculation and 82 (80%) reported enough parameters to replicate the calculation. These figures were close to the articles with significant results: 56 articles (50%) of the 112 articles reported all the required parameters and 82 (73%) reported enough parameters to replicate the calculation. We think the assertion that sample size calculation reporting is important only for trials with unsignificant results may introduce a confusion. It may imply that readers could use the pre-experiment parameters to interpret the results which is not statistically correct in the frequentist approach of the interpretation of trials1. It may also deal with post hoc power calculation which is an “ill-advised exercice” since it answers to the already answered question of the underpowering of the trials with unsignificant results2. 1. Goodman SN, Berlin JA. The use of predicted confidence intervals when planning experiments and the misuse of power when interpreting results. Ann Intern Med 1994;121(3):200-6. 2. Schulz KF, Grimes DA. Sample size calculations in randomised trials: mandatory and mystical. Lancet 2005;365(9467):1348-53. Competing interests: None declared |
|||
|
|
|||
|
Paul AVILLACH, MD, PhD student LESIM, ISPED, Université Victor Segalen Bordeaux 2, 146 rue Léo-Saignat, F-33076 Bordeaux, France, L Rachid Salmi
Send response to journal:
|
Charles et al1 illustrate clearly that sample size calculations are inadequately reported in reports of randomised control trials. This requirement applies to all research and study designs, including preclinical experiments that are often the basis of hypotheses to be tested in later clinical research. Using an approach similar to that of Charles et al, we illustrate that the situation is even worse in animal studies, where the central concept of substantive difference is seldom considered. Using Pubmed in the MEDLINE database with the equation ‘(mice OR mouse OR rat OR guinea pig) AND experiment* AND Comparative study[Mesh]’ with Limits: animal, English et French, for year 2004, we identified 2710 citations. Among the first 150 articles available as online full text, 121 (81%) reported statistically significant results. None of the 150 articles had an indication of sample size calculation. Even the randomisation process was indicated in only five articles (3%). Substantive differences and lack of power were never discussed in articles without statistical significance. Small sample sizes in animal studies are mainly related to cost and technical constraints. Our survey illustrate that the adequacy of this size to test specific hypotheses is not considered. The impact of the resulting lack of power as missed opportunities for clinical research remains to be explored. Reference 1 Charles P, Giraudeau B, Dechartres A, Baron G, Ravaud P. Reporting of sample size calculation in randomised controlled trials: review. BMJ 2009;338:b1732. Competing interests: None declared |
|||
|
|
|||
|
Jonathan D. Mayer, Professor, Epidemiology University of Washington, Seattle WA 98195 USA
Send response to journal:
|
This article addresses an extremely important topic. As a reviewer in several study sections at NIH, I have noted that it is impossible to replicate sample size and power calculations in the overwhelming number of otherwise excellent research proposals. Having noticed this, I similarly note that replication is very difficult in journal articles. The sources of sample size calculations, with reference either to a specific program and algorithm that is used, or by specifying the equations and method of calculation, are virtually absent from the literature. Rather, the norm is to state that "the study has 80% power to detect a 10% difference with an alpha of .05." The preceding is a hypothetical example. It is very rare for authors or proposal applicants to specify "using the method of Smith and Jones" or "using the sample size calculator available at ____" and then specifying a URL or statistical package. Thus, as a reviewer, it is impossible to ascertain what method was used, and the sample size specifications remain only assertions. It is vitally important that researchers refer to a specific method, statistical module, or equation. In the absence of this, readers are essentially being asked to "suspend disbelief" and assume that the authors have done the calculations correctly, and in the only way feasible. The assumption can be dangerous, and while suspension of disbelief is a wonderful tool for reading novels, it is completely inappropriate in science. We need to do better. Competing interests: None declared |
|||