Re: PACE trial authors’ reply to letter by Kindlon
Sean Lynch wrote that "protocol changes are not that unusual in large trials and can reflect pragmatic as well as scientific and ethical changes and are not necessarily sinister or representative of a hidden agenda."[1] Peter Kemp questioned whether all these protocol changes were valid or justified.[2] Lynch replied that he feels he can "trust the independence and integrity of the Steering Committee and statistical analysis".[3] However, it appears that the Trial Steering Committee (TSC) were possibly not involved with some major post-hoc changes and that the justification for at least one of these major changes was erroneous.
The Statistical Analysis Plan (SAP) was to explain protocol changes and "make public and to report in detail the planned analyses that were approved by the Trial Steering Committee in May 2010".[4] Despite impressions in the 2011 Lancet paper[5a] and authors' reply[5b] that protocol changes were guided by the SAP, approved by the TSC, and made before outcome data were examined, the SAP mentions nothing of the controversial changes which appeared in the Lancet paper, the so-called 'normal range' in fatigue and physical function, therein described as a post-hoc analysis. Nor does the SAP mention anything about the redefining of 'recovery', which is now fundamentally based on the same post-hoc 'normal range', where the recovery threshold for SF-36 physical function (PF) was lowered from 85 to 60 out of 100.[6]
White et al. of the PACE Trial claimed that the original PF threshold was abandoned because a score of >=85 points would have excluded approximately half the general working age population.[6] As outlined by Adrian Baldwin's letter, this justification was incorrect and based on confusing a mean score for a median score.[7] How did this error pass the trial statistician, independent scrutiny from the TSC, and the peer-review process of the journal in which it was published? It is unclear how this change could "more accurately reflect recovery" as claimed by PACE[6], when it is lower than the trial eligibility criteria for 'significant disability'[5] and was derived from a general population sample including the elderly and chronically disabled.[8]
PACE/White et al never justified why it was appropriate to use mean minus 1 standard deviation (S.D.) on such a dataset to calculate a threshold of normal/recovered physical function for healthy people of middle age. A previous paper co-authored by White, highlighted concerns about the use of mean minus SD on a non-normal distribution of scores[9] but PACE went on to do exactly that, despite a warning from the authors of the general population study that the distribution was heavily skewed.[8] Furthermore, the claim that their method and threshold was more "conservative"[6] than the previous work of Knoop et al.[9] (in which White was a co-author) was also incorrect. This latter paper used mean minus 1SD just as PACE did (not mean minus 2SD as claimed by PACE) and used a healthy population instead of a general population to calculate a higher threshold of 80 points. PACE maintain that a UK general population is "demographically representative".[6]
In order to avoid a “trivial" difference[10a] between the entry criteria and measures of improvement, the threshold for a 'positive outcome' in PF (also later abandoned) was previously raised from 70 to 75 points, because the entry criteria had already been raised from 60 to 65 to increase recruitment, so a 10 point gap between entry criteria and 'positive outcome' scores was successfully requested by Professor White.[10b]
Instead, the threshold for 'normal' physical function ended up being a 5 point gap in the other direction. A FOI request revealed that 13% of significantly disabled trial participants had qualified for 'normal' physical function at the beginning of the trial.[11] The mean(SD) age of trial participants at 52 week followup was about 39(12) years. To score 60 points in physical function (range 0-100, 5 point increments), a patient must report significant limitations in multiple domains (somewhere between minor limitations for 8/10 questions or major limitations in 4/10 questions), which is unusual for healthy people of working age and an unsuitable goalpost for a full 'recovery'.[12]
A study with 23 female CFS patients of similar age to PACE participants reported a mean(SD) score in physical function for the 25 healthy age-matched sedentary controls of 96.8(4.5) points.[13] A mean score of 96 was reported in a larger CFS study with 99 healthy controls[14] and a mean(SD) of 96.6(10) has also been reported for a healthy reference group consisting of 906 people.[15] The mean minus 1SD score on these (non-UK) populations, rounded up to the nearest 5 point increment, gives a threshold of 90-95 points, even higher than in the original PACE protocol before 85 was lowered to 60.
The PACE principal investigators have championed CBT/GET for CFS throughout their careers and declared to the Lancet conflicts of interest with the insurance industry.[5a] The PACE Trial demonstrated that the benefits of CBT/GET are much less than originally anticipated.[16] As an open-label trial the researchers were also not blinded to general impressions about how the trial was going.
The 2013 Psychological Medicine paper on 'recovery' indicated that changes were made to the definition before analysis was done for that particular paper, but failed to describe the 'normal range' base criteria as post-hoc, and unlike the Lancet paper did not mention TSC approval for the changes. There was no guarantee that changes occurred before examining the same data already used in previous papers. It is therefore plausible that PACE analyzed some trial data before making major, questionable, possibly unapproved, post-hoc changes to the protocol.
The 2007 PACE protocol states, "The final trial publication will include all items recommended under CONSORT."[15] In a parliamentary discussion, Baroness Wilcox stated that the MRC requires all MRC-funded clinical trials (such as PACE) comply with the CONSORT guidelines.[17] Is it generally OK to make and publish major changes to the protocol after seeing the trial outcomes data? The CONSORT Group do not think so.[18]
4. Walwyn R, Potts L, McCrone P, Johnson AL, Decesare JC, Baber H, Goldsmith K, Sharpe M, Chalder T, White PD. Trials. 2013 Nov 13;14(1):386. [Epub ahead of print] doi:10.1186/1745-6215-14-386. PMID: 24225069. http://www.trialsjournal.com/content/pdf/1745-6215-14-386.pdf
5a. White PD, Goldsmith KA, Johnson AL, Potts L, Walwyn R, DeCesare JC, Baber HL, Burgess M, Clark LV, Cox DL, Bavinton J, Angus BJ, Murphy G, Murphy M, O'Dowd H, Wilks D, McCrone P, Chalder T, Sharpe M; [PACE trial management group]. Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomised trial. Lancet. 2011 Mar 5;377(9768):823-36. Epub 2011 Feb 18. PMID: 21334061. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3065633
5b. White PD, Goldsmith KA, Johnson AL, Walwyn R, Baber HL, Chalder T, Sharpe M, [on behalf of the coauthors]. The PACE trial in chronic fatigue syndrome — Authors' reply. The Lancet, Volume 377, Issue 9780, Pages 1834 - 1835, 28 May 2011 (Published Online: 17 May 2011). doi:10.1016/S0140-6736(11)60651-X http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60651-X/fulltext
6. White PD, Goldsmith K, Johnson AL, Chalder T, Sharpe M. Recovery from chronic fatigue syndrome after treatments given in the PACE trial. Psychol Med. 2013 Oct;43(10):2227-35. doi: 10.1017/S0033291713000020. PMID: 23363640. http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid...
7. Adrian Baldwin (2013) . Re: PACE trial authors’ reply to letter by Kindlon (PACE trial steering group fail to spot error in reasons for protocol changes). BMJ Rapid Response. 21 November 2013 http://www.bmj.com/content/347/bmj.f5963/rr/673502
8. Bowling A, Bond M, Jenkinson C, Lamping DL. Short Form 36 (SF-36) Health Survey questionnaire: which normative data should be used? Comparisons between the norms provided by the Omnibus Survey in Britain, the Health Survey for England and the Oxford Healthy Life Survey. J Public Health Med. 1999 Sep;21(3):255-70. PMID 10528952. http://jpubhealth.oxfordjournals.org/content/21/3/255.full.pdf
9. Knoop H, Bleijenberg G, Gielissen MF, van der Meer JW, White PD. Is a full recovery possible after cognitive behavioural therapy for chronic fatigue syndrome? Psychother Psychosom. 2007;76(3):171-6. PMID 17426416. http://www.cfids-cab.org/rc/Knoop-1.pdf
13. VanNess JM, Stevens SR, Bateman L, Stiles TL, Snell CR. Postexertional malaise in women with chronic fatigue syndrome. J Womens Health (Larchmt). 2010 Feb;19(2):239-44. PMID 20095909.
14. Buchwald D, Pearlman T, Umali J, Schmaling K, Katon W. Functional status in patients with chronic fatigue syndrome, other fatiguing illnesses, and healthy individuals. Am J Med. 1996 Oct;101(4):364-70. PMID: 8873506.
15. Juenger J, Schellberg D, Kraemer S, Haunstetter A, Zugck C, Herzog W, Haass M. Health related quality of life in patients with congestive heart failure: comparison with other chronic diseases and relation to functional variables. Heart. 2002 Mar;87(3):235-41. PMID: 1184716.
16. White PD, Sharpe MC, Chalder T, DeCesare JC, Walwyn R; [PACE trial group]. Protocol for the PACE trial: a randomised controlled trial of adaptive pacing, cognitive behaviour therapy, and graded exercise, as supplements to standardised specialist medical care versus standardised specialist medical care alone for patients with the chronic fatigue syndrome/myalgic encephalomyelitis or encephalopathy. BMC Neurol. 2007 Mar 8;7:6. PMID 17397525. http://www.biomedcentral.com/1471-2377/7/6
Rapid Response:
Re: PACE trial authors’ reply to letter by Kindlon
Sean Lynch wrote that "protocol changes are not that unusual in large trials and can reflect pragmatic as well as scientific and ethical changes and are not necessarily sinister or representative of a hidden agenda."[1] Peter Kemp questioned whether all these protocol changes were valid or justified.[2] Lynch replied that he feels he can "trust the independence and integrity of the Steering Committee and statistical analysis".[3] However, it appears that the Trial Steering Committee (TSC) were possibly not involved with some major post-hoc changes and that the justification for at least one of these major changes was erroneous.
The Statistical Analysis Plan (SAP) was to explain protocol changes and "make public and to report in detail the planned analyses that were approved by the Trial Steering Committee in May 2010".[4] Despite impressions in the 2011 Lancet paper[5a] and authors' reply[5b] that protocol changes were guided by the SAP, approved by the TSC, and made before outcome data were examined, the SAP mentions nothing of the controversial changes which appeared in the Lancet paper, the so-called 'normal range' in fatigue and physical function, therein described as a post-hoc analysis. Nor does the SAP mention anything about the redefining of 'recovery', which is now fundamentally based on the same post-hoc 'normal range', where the recovery threshold for SF-36 physical function (PF) was lowered from 85 to 60 out of 100.[6]
White et al. of the PACE Trial claimed that the original PF threshold was abandoned because a score of >=85 points would have excluded approximately half the general working age population.[6] As outlined by Adrian Baldwin's letter, this justification was incorrect and based on confusing a mean score for a median score.[7] How did this error pass the trial statistician, independent scrutiny from the TSC, and the peer-review process of the journal in which it was published? It is unclear how this change could "more accurately reflect recovery" as claimed by PACE[6], when it is lower than the trial eligibility criteria for 'significant disability'[5] and was derived from a general population sample including the elderly and chronically disabled.[8]
PACE/White et al never justified why it was appropriate to use mean minus 1 standard deviation (S.D.) on such a dataset to calculate a threshold of normal/recovered physical function for healthy people of middle age. A previous paper co-authored by White, highlighted concerns about the use of mean minus SD on a non-normal distribution of scores[9] but PACE went on to do exactly that, despite a warning from the authors of the general population study that the distribution was heavily skewed.[8] Furthermore, the claim that their method and threshold was more "conservative"[6] than the previous work of Knoop et al.[9] (in which White was a co-author) was also incorrect. This latter paper used mean minus 1SD just as PACE did (not mean minus 2SD as claimed by PACE) and used a healthy population instead of a general population to calculate a higher threshold of 80 points. PACE maintain that a UK general population is "demographically representative".[6]
In order to avoid a “trivial" difference[10a] between the entry criteria and measures of improvement, the threshold for a 'positive outcome' in PF (also later abandoned) was previously raised from 70 to 75 points, because the entry criteria had already been raised from 60 to 65 to increase recruitment, so a 10 point gap between entry criteria and 'positive outcome' scores was successfully requested by Professor White.[10b]
Instead, the threshold for 'normal' physical function ended up being a 5 point gap in the other direction. A FOI request revealed that 13% of significantly disabled trial participants had qualified for 'normal' physical function at the beginning of the trial.[11] The mean(SD) age of trial participants at 52 week followup was about 39(12) years. To score 60 points in physical function (range 0-100, 5 point increments), a patient must report significant limitations in multiple domains (somewhere between minor limitations for 8/10 questions or major limitations in 4/10 questions), which is unusual for healthy people of working age and an unsuitable goalpost for a full 'recovery'.[12]
A study with 23 female CFS patients of similar age to PACE participants reported a mean(SD) score in physical function for the 25 healthy age-matched sedentary controls of 96.8(4.5) points.[13] A mean score of 96 was reported in a larger CFS study with 99 healthy controls[14] and a mean(SD) of 96.6(10) has also been reported for a healthy reference group consisting of 906 people.[15] The mean minus 1SD score on these (non-UK) populations, rounded up to the nearest 5 point increment, gives a threshold of 90-95 points, even higher than in the original PACE protocol before 85 was lowered to 60.
The PACE principal investigators have championed CBT/GET for CFS throughout their careers and declared to the Lancet conflicts of interest with the insurance industry.[5a] The PACE Trial demonstrated that the benefits of CBT/GET are much less than originally anticipated.[16] As an open-label trial the researchers were also not blinded to general impressions about how the trial was going.
The 2013 Psychological Medicine paper on 'recovery' indicated that changes were made to the definition before analysis was done for that particular paper, but failed to describe the 'normal range' base criteria as post-hoc, and unlike the Lancet paper did not mention TSC approval for the changes. There was no guarantee that changes occurred before examining the same data already used in previous papers. It is therefore plausible that PACE analyzed some trial data before making major, questionable, possibly unapproved, post-hoc changes to the protocol.
The 2007 PACE protocol states, "The final trial publication will include all items recommended under CONSORT."[15] In a parliamentary discussion, Baroness Wilcox stated that the MRC requires all MRC-funded clinical trials (such as PACE) comply with the CONSORT guidelines.[17] Is it generally OK to make and publish major changes to the protocol after seeing the trial outcomes data? The CONSORT Group do not think so.[18]
REFERENCES
1. Lynch S (2013). Re: PACE trial authors’ reply to letter by Kindlon. BMJ Rapid Response. http://www.bmj.com/content/347/bmj.f5963/rr/671093
2. Kemp PF (2013) Re: PACE trial authors’ reply to letter by Kindlon BMJ Rapid Response. http://www.bmj.com/content/347/bmj.f5963/rr/671436
3. Lynch S (2013). Re: PACE trial authors’ reply to letter by Kindlon. BMJ Rapid Response. http://www.bmj.com/content/347/bmj.f5963/rr/671597
4. Walwyn R, Potts L, McCrone P, Johnson AL, Decesare JC, Baber H, Goldsmith K, Sharpe M, Chalder T, White PD. Trials. 2013 Nov 13;14(1):386. [Epub ahead of print] doi:10.1186/1745-6215-14-386. PMID: 24225069. http://www.trialsjournal.com/content/pdf/1745-6215-14-386.pdf
5a. White PD, Goldsmith KA, Johnson AL, Potts L, Walwyn R, DeCesare JC, Baber HL, Burgess M, Clark LV, Cox DL, Bavinton J, Angus BJ, Murphy G, Murphy M, O'Dowd H, Wilks D, McCrone P, Chalder T, Sharpe M; [PACE trial management group]. Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomised trial. Lancet. 2011 Mar 5;377(9768):823-36. Epub 2011 Feb 18. PMID: 21334061. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3065633
5b. White PD, Goldsmith KA, Johnson AL, Walwyn R, Baber HL, Chalder T, Sharpe M, [on behalf of the coauthors]. The PACE trial in chronic fatigue syndrome — Authors' reply. The Lancet, Volume 377, Issue 9780, Pages 1834 - 1835, 28 May 2011 (Published Online: 17 May 2011). doi:10.1016/S0140-6736(11)60651-X http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60651-X/fulltext
6. White PD, Goldsmith K, Johnson AL, Chalder T, Sharpe M. Recovery from chronic fatigue syndrome after treatments given in the PACE trial. Psychol Med. 2013 Oct;43(10):2227-35. doi: 10.1017/S0033291713000020. PMID: 23363640. http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid...
7. Adrian Baldwin (2013) . Re: PACE trial authors’ reply to letter by Kindlon (PACE trial steering group fail to spot error in reasons for protocol changes). BMJ Rapid Response. 21 November 2013 http://www.bmj.com/content/347/bmj.f5963/rr/673502
8. Bowling A, Bond M, Jenkinson C, Lamping DL. Short Form 36 (SF-36) Health Survey questionnaire: which normative data should be used? Comparisons between the norms provided by the Omnibus Survey in Britain, the Health Survey for England and the Oxford Healthy Life Survey. J Public Health Med. 1999 Sep;21(3):255-70. PMID 10528952. http://jpubhealth.oxfordjournals.org/content/21/3/255.full.pdf
9. Knoop H, Bleijenberg G, Gielissen MF, van der Meer JW, White PD. Is a full recovery possible after cognitive behavioural therapy for chronic fatigue syndrome? Psychother Psychosom. 2007;76(3):171-6. PMID 17426416. http://www.cfids-cab.org/rc/Knoop-1.pdf
10a. https://listserv.nodak.edu/cgi-bin/wa.exe?A2=ind1301b&L=co-cure&F=&S=&P=...
10b. http://www.ico.org.uk/~/media/documents/decisionnotices/2013/fs_50463661...
11. Queen Mary, University of London. FOI Request: 2013/F42. https://docs.google.com/document/d/17H8AQsMKkXnKDdm0JbpAWQXWWTNEXvaKWNRO... (Accessed: 24th November 2013).
12. http://www.rand.org/health/surveys_tools/mos/mos_core_36item_survey.html
13. VanNess JM, Stevens SR, Bateman L, Stiles TL, Snell CR. Postexertional malaise in women with chronic fatigue syndrome. J Womens Health (Larchmt). 2010 Feb;19(2):239-44. PMID 20095909.
14. Buchwald D, Pearlman T, Umali J, Schmaling K, Katon W. Functional status in patients with chronic fatigue syndrome, other fatiguing illnesses, and healthy individuals. Am J Med. 1996 Oct;101(4):364-70. PMID: 8873506.
15. Juenger J, Schellberg D, Kraemer S, Haunstetter A, Zugck C, Herzog W, Haass M. Health related quality of life in patients with congestive heart failure: comparison with other chronic diseases and relation to functional variables. Heart. 2002 Mar;87(3):235-41. PMID: 1184716.
16. White PD, Sharpe MC, Chalder T, DeCesare JC, Walwyn R; [PACE trial group]. Protocol for the PACE trial: a randomised controlled trial of adaptive pacing, cognitive behaviour therapy, and graded exercise, as supplements to standardised specialist medical care versus standardised specialist medical care alone for patients with the chronic fatigue syndrome/myalgic encephalomyelitis or encephalopathy. BMC Neurol. 2007 Mar 8;7:6. PMID 17397525. http://www.biomedcentral.com/1471-2377/7/6
17. Parliamentary Questions | Controlled trials, in particular the PACE trial | 22 May 2012. http://www.meassociation.org.uk/2012/05/parliamentary-questions-controll...
18. http://www.consort-statement.org
Competing interests: No competing interests