Intention-to-treat analysis was over-zealous - but this does not affect findings
I would like to comment on the extraction of data from published
reports in this meta-analysis. Beich et al [1] correctly aimed to perform
intention-to-treat analysis by regarding all missing outcomes as treatment
failures, but I question the implementation of the intention-to-treat
analysis.
The data extraction for the trial by Richmond et al [2] has been
criticised on two grounds. First, the extracted data differ from the
results in the primary publication [3]. This is entirely legitimate,
because the primary outcome in the meta-analysis was the proportion of
successes at follow-up, while Richmond et al's published result was
largely driven by a baseline imbalance [4]. Second, in extracting the
data, Beich et al made a correction for missing outcomes, ignoring the
fact that the published analysis had already made such a correction [5].
Correcting this error changes Beich et al's results of 16/96
(intervention) vs. 13/93 (control) to 23/96 vs. 20/93, and reduces the
risk difference from 2.7% to 2.5%. This error is unimportant, but
unfortunately Beich et al have not acknowledged it.
On reading the reports of the other trials, I found two further
errors, also resulting from incorrect handling of missing data.
The trial by Wallace et al [6] also reported an analysis using an
intention-to-treat approach in which those not interviewed were assumed to
be still drinking excessively. For example, in the treated arm, 448 men
and women were randomised. At 12 months, 247 were considered to be
drinking excessively and 201 were not. It is possible to work out that the
247 excessive drinkers include 85 men and women who were not interviewed.
Beich et al, however, give the number of successes as 116, which can only
be calculated by wrongly subtracting the number of missing outcomes (85)
from the correct number of successes (201). Correcting this error changes
the results from Beich et al's 116/448 vs. 48/459 to 201/448 vs. 122/459,
and increases the risk difference for this trial from 15.4% to 18.3%.
The trial by Ockene et al [7] only reported the numbers of successes
for the subgroup with excessive weekly drinking at baseline; further, it
excluded individuals with missing outcome. In assigning individuals with
missing outcomes as failures, Beich et al have used all those randomised
as a denominator. This implicitly treats all those not drinking
excessively at baseline as failures, which is unreasonable. It is not
possible to get a correct analysis, because Ockene et al do not report
total numbers drinking excessively at baseline. An improvement on Beich et
al's approach is to exclude from the denominator all those with observed
outcome and non-excessive weekly drinking at baseline, but to include all
those with missing outcome. This changes the results from Beich et al's
102/274 vs. 66/256 to 102/216 vs. 66/193, and increases the risk
difference for this trial from 11.4% to 13.0%.
It is important to note that these over-zealous applications of
intention-to-treat analysis have little effect on the results of the meta-
analysis. Correcting all three errors increases the intervention effect
from 10.5% to 11.5%. The numerical findings of the meta-analysis should
therefore be accepted.
References
1 Beich A, Thorsen T, Rollnick S. Screening in brief intervention
trials targeting excessive drinkers in general practice: systematic review
and meta-analysis. BMJ 2003;327:536-542.
2 Richmond R, Heather N, Wodak A, Kehoe L, Webster I. Controlled
evaluation of a general practice-based brief intervention for excessive
drinking. Addiction 1995;90:119-32.
3 Heather N, Richmond R. Screening effect ten times greater than
calculated by Beich et al. Rapid response bmj.com; 1 October 2003.
4 Beich A, Thorsen T. Re: Screening effect ten times greater (by
Heather and Richmond). Rapid response bmj.com; 5 October 2003.
5 Heather N. Beich et al have got it wrong again. Rapid response
bmj.com; 24 October 2003.
6 Wallace P, Cutler S, Haines A. Randomised controlled trial of
general practitioner intervention in patients with excessive alcohol
consumption. BMJ 1988;297:663-8.
7 Ockene JK, Adams A, Hurley TG, Wheeler EV, Hebert JR. Brief
physician- and nurse practitioner-delivered counseling for high-risk
drinkers: does it work? Arch Intern Med 1999;159:2198-205.
Competing interests:
I have received a fee from the Health Development Agency for writing a review of this meta-analysis.
Rapid Response:
Intention-to-treat analysis was over-zealous - but this does not affect findings
I would like to comment on the extraction of data from published
reports in this meta-analysis. Beich et al [1] correctly aimed to perform
intention-to-treat analysis by regarding all missing outcomes as treatment
failures, but I question the implementation of the intention-to-treat
analysis.
The data extraction for the trial by Richmond et al [2] has been
criticised on two grounds. First, the extracted data differ from the
results in the primary publication [3]. This is entirely legitimate,
because the primary outcome in the meta-analysis was the proportion of
successes at follow-up, while Richmond et al's published result was
largely driven by a baseline imbalance [4]. Second, in extracting the
data, Beich et al made a correction for missing outcomes, ignoring the
fact that the published analysis had already made such a correction [5].
Correcting this error changes Beich et al's results of 16/96
(intervention) vs. 13/93 (control) to 23/96 vs. 20/93, and reduces the
risk difference from 2.7% to 2.5%. This error is unimportant, but
unfortunately Beich et al have not acknowledged it.
On reading the reports of the other trials, I found two further
errors, also resulting from incorrect handling of missing data.
The trial by Wallace et al [6] also reported an analysis using an
intention-to-treat approach in which those not interviewed were assumed to
be still drinking excessively. For example, in the treated arm, 448 men
and women were randomised. At 12 months, 247 were considered to be
drinking excessively and 201 were not. It is possible to work out that the
247 excessive drinkers include 85 men and women who were not interviewed.
Beich et al, however, give the number of successes as 116, which can only
be calculated by wrongly subtracting the number of missing outcomes (85)
from the correct number of successes (201). Correcting this error changes
the results from Beich et al's 116/448 vs. 48/459 to 201/448 vs. 122/459,
and increases the risk difference for this trial from 15.4% to 18.3%.
The trial by Ockene et al [7] only reported the numbers of successes
for the subgroup with excessive weekly drinking at baseline; further, it
excluded individuals with missing outcome. In assigning individuals with
missing outcomes as failures, Beich et al have used all those randomised
as a denominator. This implicitly treats all those not drinking
excessively at baseline as failures, which is unreasonable. It is not
possible to get a correct analysis, because Ockene et al do not report
total numbers drinking excessively at baseline. An improvement on Beich et
al's approach is to exclude from the denominator all those with observed
outcome and non-excessive weekly drinking at baseline, but to include all
those with missing outcome. This changes the results from Beich et al's
102/274 vs. 66/256 to 102/216 vs. 66/193, and increases the risk
difference for this trial from 11.4% to 13.0%.
It is important to note that these over-zealous applications of
intention-to-treat analysis have little effect on the results of the meta-
analysis. Correcting all three errors increases the intervention effect
from 10.5% to 11.5%. The numerical findings of the meta-analysis should
therefore be accepted.
References
1 Beich A, Thorsen T, Rollnick S. Screening in brief intervention
trials targeting excessive drinkers in general practice: systematic review
and meta-analysis. BMJ 2003;327:536-542.
2 Richmond R, Heather N, Wodak A, Kehoe L, Webster I. Controlled
evaluation of a general practice-based brief intervention for excessive
drinking. Addiction 1995;90:119-32.
3 Heather N, Richmond R. Screening effect ten times greater than
calculated by Beich et al. Rapid response bmj.com; 1 October 2003.
4 Beich A, Thorsen T. Re: Screening effect ten times greater (by
Heather and Richmond). Rapid response bmj.com; 5 October 2003.
5 Heather N. Beich et al have got it wrong again. Rapid response
bmj.com; 24 October 2003.
6 Wallace P, Cutler S, Haines A. Randomised controlled trial of
general practitioner intervention in patients with excessive alcohol
consumption. BMJ 1988;297:663-8.
7 Ockene JK, Adams A, Hurley TG, Wheeler EV, Hebert JR. Brief
physician- and nurse practitioner-delivered counseling for high-risk
drinkers: does it work? Arch Intern Med 1999;159:2198-205.
Competing interests:
I have received a fee from the Health Development Agency for writing a review of this meta-analysis.
Competing interests: No competing interests