Study was much too small for inferences to be drawn

EDITOR—Inconclusive genetic studies should not be published in the BMJ. Once again the journal has published a genetic association study based on a tiny sample of cases and producing scarcely significant results.1 It has allowed the authors to claim that they may have identified an important risk factor for a common and serious disease, on the basis of only the flimsiest of evidence.

The authors report a study that, owing to its prospective design, was of just 27 subjects who developed diabetes along with 481 others who remained unaffected. They identified the C282Y mutation in the haemochromatosis gene as a predictor for developing diabetes (P=0.047). Given the very small numbers in the study (with only five subjects carrying the mutation becoming affected), it seems implausible that this P value is even accurate, but it is certainly not small enough to provide any real evidence for this polymorphism being a risk factor for diabetes.

The small number of affected subjects further weakens the conclusions because, as I have argued elsewhere,2 results based on small samples are more likely to represent type 1 errors than results from large samples which produce an equivalent P value. Because genetic risk factors are continuously present throughout life, prospective studies have relatively few advantages; case-control designs are usually to be preferred because of the large number of affected subjects they provide. For a disease as common as diabetes it is ludicrous to draw inferences from only 27 affected subjects.

Publishing this kind of study without appropriately cautious discussion can have a pernicious effect. For example, the authors have been allowed to claim that “the C282Y mutation also predisposes to coronary heart disease.” There is practically no evidence to support this. The authors cite a paper of their own,3 which is in fact very similar to the current one and reports similarly unimpressive P values. Once such statements are published many readers will naively accept them rather than being aware that they are in fact highly speculative.

It is ironic that a paper that would have struggled to get into any respectable genetics journal should have been published in a general journal with a high profile and a readership less familiar with the caveats that should apply to this kind of study. The BMJ does medicine a great disservice by giving undue prominence to such inconclusive and flawed work.

Authors' reply

EDITOR—Curtis criticises our work showing an association between the C282Y mutation in the haemochromatosis gene and the incidence of type 2 diabetes in 508 men, who were followed up for four years. During follow up 27 men developed diabetes. As we discussed in our paper, this study is small. Consequently, the statistical certainty of the estimate of relative risk (3.5) is rather low, as indicated by the wide confidence interval. The way to proceed in science is that our findings will be retested in further studies. In fact, we will complete an 11 year follow up of our cohort by the end of this year. This will allow us to retest our hypothesis with a larger number of incident cases.

Curtis claims that case-control studies would be preferable in genetic epidemiology. We disagree. Most retrospective case-control studies are flawed by the survival and incidence-prevalence biases. Unfortunately, prospective studies are few, but those few have shown that the results from prospective and retrospective studies tend to differ. We believe that it is more important to avoid the biases of retrospective studies than to attempt to enhance power by collecting a large number of cases retrospectively. We consider retrospective case-control studies to be virtually worthless in genetic epidemiology and wish that editors and readers would pay more attention to the study design and the validity of the findings than to large numbers.