Letters

Rapid Responses from bmj.com

The use of masking procedures

Separating results and conclusions

Authors' response

From BMJ USA 2002;October:570

As of September 19, 2002, this article had generated 8 Rapid Responses, which can be read in their entirety at http://bmj.com/cgi/eletters/325/7358/249. Edited excerpts of two responses, and the author's reply, are presented here.Editor

The use of masking procedures

EDITORAlthough there seems to be good agreement in this study between raters' assessment of authors' conclusions on a 6-point scale, there is enough variation and subjectivity in this assessment for bias to be a serious concern. At the very least, assessments of authors' conclusions should be made without knowledge of the source of funding, and assessment of the source of funding should be made without knowledge of the authors' conclusions.

The procedures for masking or blinding in this investigation do not meet this standard. One investigator assessed both source of funding and authors' conclusions without any masking. The second investigator assessed authors' conclusions without knowledge of the source of funding. Unfortunately, the two investigators then colluded together to agree on a final score, thus diluting the benefit of masking. Even if they had not colluded and the unblinded investigator's assessment of authors' conclusions had not been used, the possibility of bias remains because the only assessment of source of funding was made with knowledge of the authors' conclusions.

William P Plummer, consultant psychiatrist. 
East Kent Community NHS Trust, Canterbury, UK wpplummer{at}aol.com

Separating results and conclusions

EDITORKjaergard and Als-Nielsen present interesting data on the relationship between competing interests and conclusions, but I think they have missed an opportunity to tell us something even more interesting about the reasons for that relationship.

The observed association could be because authors of industry-sponsored publications are more likely to draw inappropriately positive conclusions, or it could be because industry-sponsored research is more likely to reach positive results. This could be for many reasons, perhaps because pharmaceutical companies are more likely to fund trials if they believe the results are likely to be positive. Or perhaps industry-sponsored studies, usually being better funded than independent studies, are more likely to be adequately powered and therefore less likely to reach a false-negative conclusion. The difference between these potential explanations is important.

The authors' attempts to address this by statistical correction for sample size seem crude. What is important is not the sample size of the study, but its power to detect a difference between treatments. A study with 200 patients that only needed 150 to have 90% power to detect a clinically important difference between treatments is clearly a more highly powered study than one with 500 patients that should have had 1000, but this would not be captured by the analysis used in the paper.

Adam Jacobs, director. 
Dianthus Medical Limited, London, UK ajacobs{at}dianthus.co.uk

Authors' response

EDITORWe assessed authors' conclusions without and with blinding of competing interests. Consensus was achieved in all cases before the analyses, and blinding was maintained until after completion of the analyses. Reliability analyses showed good agreement between the unblinded and blinded assessment of authors' conclusions. Unlike the assessment of authors' conclusions, the extraction of competing interests was not based on subjective assessment but on authors' reporting of financial or other competing interests. Incorrect registration of this information could have occurred by mistake or intentionally. We estimated that unintentional mistakes would not be affected by blinding and that we would not make intentional mistakes. Accordingly, we did not blind the extraction of competing interests.

To comply with Plummer's critique, we analyzed the association between competing interests and authors' conclusions assessed in a blinded manner. This confirmed that competing interests were significantly associated with authors' conclusions (Kruskal-Wallis H=17.25 with 3 degrees of freedom; P=0.001), suggesting that the risk of bias is not large enough to dismiss the results of our study. We acknowledge that larger studies are needed to confirm our results and to estimate the generalizability of our findings.

In response to Jacobs, we recognize that the statistical power of trials depends on the number of included patients and events. From each of the 159 trials, we extracted the number of included patients, authors' sample size calculations (reflecting the expected event rate), and whether the required sample size was reached. The required sample size was not attained in 20 (13%) trials. None of these trials underwent preliminary termination due to differences between outcomes in experimental and control groups. These potential confounding factors did not explain the association between competing interests and authors' conclusions. Accordingly, the association between competing interests and authors' conclusions does not seem to reflect differences in statistical power.

Lise L Kjaergard, consultant physician. 
Cochrane Hepato-Biliary Group, Copenhagen Trial Unit, Rigshospitalet, Denmark kjaergard{at}ctu.rh.dk