Randomised controlled trial of butterbur and cetirizine for treating seasonal allergic rhinitis

We would like to raise two points regarding the paper by Schapowal et
al. We feel there may be a possibility of bias and we would question the
statistical interpretation of the data.

This is an unusual clinical trial in that it seems to be trying to
prove no difference between the treatment arms. Most trials are trying to
show that one treatment is better than the other. It is this unusual
feature which gives us concern regarding the methodology and the analysis.

In the usual situation, where a trial is investigating a possible
difference between treatments, double blinding is used to prevent bias by
patients or researchers. Without knowing which treatment they are on, they
cannot consistently overestimate one or other of the treatment effects.
However, if the hypothesis is that there is no difference between
treatments, it is possible to bias the result towards supporting the
hypothesis. This happens if one tends to score all patients similarly.
This bias could easily result unconsciously on behalf of patients and
researchers. Given that the trial on Butterbur (a petasite) is conducted
by the Petasites study Group there may well be a strong desire for the
researchers to prove their hypothesis and this makes them vulnerable to
unconsciously biasing the results in this way.

We would also like to question the interpretation of the data. The
authors use a “Mann-Whitney rank sum test (one sided)”. They state that
significant values mean Butterbur is not inferior. The P values for this
test are mostly 0.001. This seems to imply that Butterbur has been shown
to be not inferior to cetirizine with only a 0.1% chance of this result
being wrong.

Our understanding is that in general there are two errors that a
trial can make. Type one error is for the trial to show a difference
between treatments when, in reality there is no true difference. The risk
of this error occurring is quantified by the P value. The P value is
therefore important when assessing the usual type of trial which is
claiming to show a difference. Type two error on the other hand is for the
trial to show no difference between treatments when, in reality there is a
true difference. The risk of this error occurring is quantified by the
power calculation. Therefore it is the power of the study which is crucial
in this trial which claims to have found no difference between treatments.

The authors quote an 80% power for the study. This means that there
is a 20% chance of a type two error occurring. That is a 20% chance that
there is in reality a difference between Butterbur and cetirizine and that
the trial results are wrong, (not a 0.1% chance as implied in the paper.)

To summarise, the trial fails to show evidence of a difference
between the treatments but it also fails to show evidence of no
difference. There is subtle but very important distinction between these
two statements. In effect the trial does not really give any support one
way or another to justify the use of Butturbur.