Professor Lynch remarks on changes to the PACE Trial Protocol: "…the alterations mentioned do appear well justified in my view."
I beg to differ.
The PACE Trial website FAQ2 written after the Lancet publication, answers this question: "27. Why did you change the analysis plan of the primary outcomes?
(http://www.pacetrial.org/faq/faq2.html)
(Please include the PACE author's answer as forming part of this comment)
The answer provided states: "A detailed statistical analysis plan was written, mainly by the trial statisticians, and approved by the independent Trial Steering Committee before examining the trial outcome data. This is common practice in clinical trials."
This does not explain anything. Hinting that the statisticians or Steering Committee defined the Primary Outcome Measures is misleading. Furthermore, the World Health Organization state (1):
"…once a protocol for the study has been developed and approved, and the study has started and progressed, it should be adhered to strictly and should not be changed. This is particularly important in multi-centre studies. Violations of the protocol can discredit the whole study…"
Ben Goldacre remarked in The Guardian (2) "…in a trial, you might measure many things but you have to say which is the "primary outcome" before you start: you can't change your mind about what you're counting as your main outcome after you've finished and the results are in. It's not just dodgy, it also messes with the statistics." Goldacre added, "You cannot change the rules after the game has started. You cannot even be seen to do that."
The author's complaint that the originally chosen outcomes "would be hard to interpret" does not make sense. The differences between all 4 trial arms remains proportionate whatever analysis is used and there was nothing to prevent them producing as many post hoc comparisons as they could wish, without discarding the Protocol and creating ludicrous anomalies in the process.
Replacing the "composite (two-fold) outcomes" does not explain why at the same time, they lowered the Primary Outcome target thresholds by around 30 to 50 percent and discarded 'Recovery' criteria entirely.
In October 2004 amendments were made to the Protocol including review of the Primary Outcome Measures. The full Protocol states (3):
"o Protocol version 2.1, 22 October 2004.
…We have made some minor changes to the protocol with the addition of measures in order to: properly measure meaningful outcome…" (p.79)
Including:
"o A modification to the primary outcome, by the addition of a 50% reduction in fatigue and physical disability being a positive outcome, alongside the previously approved categorical outcome.
o An additional primary outcome of needing both fatigue and physical disability to improve
o Operationalised criteria for recovery" (p.80)
Therefore the aim of the trial from its outset with its authoritatively established primary outcome measures and thresholds were specifically designed in order to "properly measure meaningful outcome". They should not have required any alteration.
The PACE Trial Protocol states: "There is therefore an urgent need to: (a) compare the supplementary therapies of both CBT and GET with both APT and standardised specialist medical care (SSMC) alone, seeking evidence of both benefit and harm (b) compare supplementary APT against SSMC alone and (c) compare the supplementary therapies of APT, CBT and GET in order to clarify differential predictors and mechanisms of change."
This could not be clearer. The trial was designed to compare the treatments with each other and the control group. Which raises the question:
If the trial was designed specifically to allow comparisons between the treatments, what led the researchers to believe that they would have to alter the Primary Outcome thresholds in order to "answer our main questions regarding comparative efficacy"?
When the Protocol was designed, the researchers analysed previous research, including their own, into GET and CBT for ME/CFS. This evidence informed the establishment of the Outcome Measure thresholds to, "properly measure meaningful outcome".
The Protocol states: "We have chosen 15 sessions for all supplementary treatments on the basis of the previous trials of CBT and GET [18,23-26], as well as extensive clinical experience." And: "The existing evidence does not allow precise estimates of improvement with the trial treatments. However the available data suggests that at one year follow up, 50 to 63% of participants with CFS/ME had a positive outcome, by intention to treat, in the three RCTs of rehabilitative CBT [18,25,26], with 69% improved after an educational rehabilitation that closely resembled CBT [43]. This compares to 18 and 63% improved in the two RCTs of GET [23,24], and 47% improvement in a clinical audit of GET [56]."
Therefore the Primary Outcome measures in the Protocol were based on prior research, including that done by the Principal Investigators themselves and on their "extensive clinical experience". The PACE author's do not explain why they then decided to lower their own expertly determined target thresholds by 30% to 50%, and neither does Professor Lynch's comment.
Rapid Response:
Re: PACE trial authors’ reply to letter by Kindlon
Reply to BEYOND PACE - MOVING THE DEBATE FORWARD
http://www.bmj.com/content/347/bmj.f5963/rr/671093
Professor Lynch remarks on changes to the PACE Trial Protocol: "…the alterations mentioned do appear well justified in my view."
I beg to differ.
The PACE Trial website FAQ2 written after the Lancet publication, answers this question: "27. Why did you change the analysis plan of the primary outcomes?
(http://www.pacetrial.org/faq/faq2.html)
(Please include the PACE author's answer as forming part of this comment)
The answer provided states: "A detailed statistical analysis plan was written, mainly by the trial statisticians, and approved by the independent Trial Steering Committee before examining the trial outcome data. This is common practice in clinical trials."
This does not explain anything. Hinting that the statisticians or Steering Committee defined the Primary Outcome Measures is misleading. Furthermore, the World Health Organization state (1):
"…once a protocol for the study has been developed and approved, and the study has started and progressed, it should be adhered to strictly and should not be changed. This is particularly important in multi-centre studies. Violations of the protocol can discredit the whole study…"
Ben Goldacre remarked in The Guardian (2) "…in a trial, you might measure many things but you have to say which is the "primary outcome" before you start: you can't change your mind about what you're counting as your main outcome after you've finished and the results are in. It's not just dodgy, it also messes with the statistics." Goldacre added, "You cannot change the rules after the game has started. You cannot even be seen to do that."
The author's complaint that the originally chosen outcomes "would be hard to interpret" does not make sense. The differences between all 4 trial arms remains proportionate whatever analysis is used and there was nothing to prevent them producing as many post hoc comparisons as they could wish, without discarding the Protocol and creating ludicrous anomalies in the process.
Replacing the "composite (two-fold) outcomes" does not explain why at the same time, they lowered the Primary Outcome target thresholds by around 30 to 50 percent and discarded 'Recovery' criteria entirely.
In October 2004 amendments were made to the Protocol including review of the Primary Outcome Measures. The full Protocol states (3):
"o Protocol version 2.1, 22 October 2004.
…We have made some minor changes to the protocol with the addition of measures in order to: properly measure meaningful outcome…" (p.79)
Including:
"o A modification to the primary outcome, by the addition of a 50% reduction in fatigue and physical disability being a positive outcome, alongside the previously approved categorical outcome.
o An additional primary outcome of needing both fatigue and physical disability to improve
o Operationalised criteria for recovery" (p.80)
Therefore the aim of the trial from its outset with its authoritatively established primary outcome measures and thresholds were specifically designed in order to "properly measure meaningful outcome". They should not have required any alteration.
The PACE Trial Protocol states: "There is therefore an urgent need to: (a) compare the supplementary therapies of both CBT and GET with both APT and standardised specialist medical care (SSMC) alone, seeking evidence of both benefit and harm (b) compare supplementary APT against SSMC alone and (c) compare the supplementary therapies of APT, CBT and GET in order to clarify differential predictors and mechanisms of change."
This could not be clearer. The trial was designed to compare the treatments with each other and the control group. Which raises the question:
If the trial was designed specifically to allow comparisons between the treatments, what led the researchers to believe that they would have to alter the Primary Outcome thresholds in order to "answer our main questions regarding comparative efficacy"?
When the Protocol was designed, the researchers analysed previous research, including their own, into GET and CBT for ME/CFS. This evidence informed the establishment of the Outcome Measure thresholds to, "properly measure meaningful outcome".
The Protocol states: "We have chosen 15 sessions for all supplementary treatments on the basis of the previous trials of CBT and GET [18,23-26], as well as extensive clinical experience." And: "The existing evidence does not allow precise estimates of improvement with the trial treatments. However the available data suggests that at one year follow up, 50 to 63% of participants with CFS/ME had a positive outcome, by intention to treat, in the three RCTs of rehabilitative CBT [18,25,26], with 69% improved after an educational rehabilitation that closely resembled CBT [43]. This compares to 18 and 63% improved in the two RCTs of GET [23,24], and 47% improvement in a clinical audit of GET [56]."
Therefore the Primary Outcome measures in the Protocol were based on prior research, including that done by the Principal Investigators themselves and on their "extensive clinical experience". The PACE author's do not explain why they then decided to lower their own expertly determined target thresholds by 30% to 50%, and neither does Professor Lynch's comment.
Peter Kemp MA
References
1. World Health Organization Online: http://whqlibdoc.who.int/emro/2004/9290213639.pdf
2. Ben Goldacre. Clinical trials and playing by the rules. The Guardian, Saturday January 5 2008
3. Online: http://www.meactionuk.org.uk/FULL-Protocol-SEARCHABLE-version.pdf
Competing interests: No competing interests