Call to action: RIAT reanalysis of the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) Study
We issue a call to action to restore the reporting of STAR*D’s six trials [1-6] and summary article reporting 12-month follow-up findings. [7]
STAR*D (NCT00021528) is the largest and most consequential antidepressant study ever conducted. Funded by the National Institute of Mental Health (NIMH) for 35-million $US, STAR*D enrolled 4,041 patients between 2001-2006 who screened positive for major depression while seeking routine medical or psychiatric care and included patients with common medical and psychiatric comorbid conditions thereby increasing the generalizability of its findings.
STAR*D provided up to 4 treatment steps per patient and was designed to give guidance in selecting the best next-step option for patients who fail to gain sufficient relief from their first, and/or subsequent, trial(s). Antidepressants were administered using a system of measurement-based care that assessed symptoms and side-effects each visit to guide aggressive medication dosing “to ensure that the likelihood of achieving remission was maximized and that those who did not reach remission were truly resistant to the medication.”[1, p. 30]
STAR*D evaluated the relative effectiveness of 11 pharmacologically distinct drug–drug combination treatments in five head-to-head comparisons. [2-6] Patients who achieved remission were encouraged to enter one-year free follow-up care, as were responder patients who failed to attain remission but did not want to continue to the next step as this involved changing medication.
STAR*D’s prespecified primary outcome was remission, defined as scoring <8 on the blindly-administered Hamilton Rating Scale of Depression (HRSD). Response was a secondary outcome defined as ≥50% HRSD reduction. STAR*D had 11 additional prespecified secondary outcome measures assessing health, work and social functioning, quality of life, side-effect burden, medical utilization and cost, and personal income. Both the prespecified primary and secondary outcomes were administered at entry into and exit from each treatment step and follow-up months 3, 6, 9, and 12. [8, table 2]
The investigators state, “STAR*D uses a randomized, controlled design to evaluate both the theoretical principles and clinical beliefs that currently guide the management of treatment-resistant depression in terms of symptoms, function, satisfaction, side-effect burden, and health care utilization and cost estimates. Given the dearth of controlled data, results should have substantial public health and scientific significance.” [9, p. 136]
Thirteen years later, STAR*D researchers have still not published steps 1-4 pre/post and follow-up findings for the secondary outcomes as prespecified. Other researchers though have conducted post-hoc analyses using a subset of the secondary outcomes typically focused on special populations. [eg, 10, 11] Elsewhere we document STAR*D researchers evidenced multiple protocol violations with each violation having an inflationary effect on reported remission and response rates. [12-14]
Given STAR*D’s “substantial public health and scientific significance,” we intend to restore the record for these trials following the principles of the Restoring Invisible and Abandoned Trials (RIAT) initiative. [15]
Our rationale for correcting the record
Apparent protocol violations include:
• Following data collection and without disclosure, substituting a non-blinded/clinic-administered assessment that was used to guide care as the secondary outcome to report remission rates and sole outcome to report response rates in steps 1‐4, and sole outcome measure to report remission and response rates in the summary article. This switch occurred despite STAR*D’s protocol specifically excluding all non-blinded/clinic-administered assessments from use as research outcome measures. [8, p. 47-48]
• Used data from 931 patients deemed ineligible for analysis in step-1 in their steps 2-4 and summary articles without clear disclosure: 607 because these patients’ <14 HRSD score signified at most only mild depressive symptoms when entering the study as well as 324 patients without a blindly-administered baseline HRSD. [1, figure 1] Furthermore, our analysis of STAR*D’s dataset discovered that 99 of the 607 patients scored <8 on their baseline HRSD—STAR*D’s remission criterion. Despite scoring as not depressed at study outset, the outcome data from many of these 99 patients were included in the steps 2-4 articles and all were included in the summary article.
• Failed to disclose that all patients started on citalopram in their baseline visit while excluding from analysis the 370 who then dropped out, [1, figure 1; 7, figure 1] despite the step-1 article stating, “our primary analyses classified patients with missing exit HRSD scores as nonremitters a priori.” [1, p. 34] These 370 early dropout patients did not have an exit HRSD and therefore should have been counted as treatment failures as prespecified.
Recently, we published a step-1 reanalysis using NIMH’s STAR*D dataset. [14] This analysis found substantial reductions in remission and response rates using the blindly-administered HRSD. While step-1 authors claimed “the response and remission rates in this highly generalizable sample with substantial axis I and axis III comorbidity closely resemble those seen in 8-week efficacy trials,” [1, p. 28] our analysis found citalopram’s actual real-world effectiveness was approximately half that reported in efficacy trials.
Surprisingly, an exploratory analysis of step-2’s dataset discovered 117 patients scoring <8 on the HRSD at entry into step-2 yet were included in the step-2 analyses. In other words, 117 patients scored as remitted prior to starting their step-2 medication yet were included in STAR*D’s medication switch [2] and augmentation [3] articles. This occurred despite STAR*D researchers prespecifying, “participants who begin a level with HRSD <8 will be excluded from analyses.” [9, p. 130]
The fact that there were no reported statistical differences in remission and response rates for any of STAR*D’s five head-to-head trials of 11 chemically-distinct medications/medication combinations—and therefore providing no evidence-based guidance for the next best medication option after a failed trial—may be due to these protocol violations versus what actually occurred had the research protocol been followed.
Scope of our restoration
Given the unreported deviations from STAR*D’s protocol, an accurate restoration using the prespecified primary and secondary outcome measures for steps 1-4 and follow-up is essential to understanding the true risks and benefits of each STAR*D treatment, and determine whether to embrace STAR*D’s investigators repeated claims of the superiority of their measurement-based, treat-to-remission, model of care.
Our sources
We will conduct a RIAT reanalysis of the patient-level data in adherence to the original protocol and statistical analysis plan obtained from the NIMH Data Archive unless STAR*D’s authors publicly commit to do so. [17]
Questions our restoration aims to answer
1. What is the fidelity between steps 1-4 published outcomes compared to those from the restored patient-level data using the prespecified outcome measures, inclusion for analysis criteria, and statistical plan?
2. What are the comparative outcomes for steps 2-4 head-to-head trials using the prespecified outcome measures, inclusion criteria, and statistical plan?
3. What are the comparative outcomes during follow-up using the prespecified outcome measures, inclusion criteria, and statistical plan?
References:
1. Trivedi MH, Rush AJ, Wisniewski SR, Nierenberg AA, Warden D, Ritz L, Norquist G, Howland RH, Lebowitz B, McGarth PJ, Shores-Wilson K, Biggs MM, Balasubramani GK, Fava M. Evaluation of outcomes with citalopram for depression using measurement-based care in STAR*D: implications for clinical practice. Am J Psychiatry. 2006;163:28–40.
2. Rush AJ, Trivedi MH, Wisniewski SR, Stewart JW, Nierenberg AA, Thase ME, Ritz L, Biggs MM, Warden D, Luther JF, Shores-Wilson K, Niederehe G, Fava M. Bupropion-SR, sertraline, or venlafaxine-XR after failure of SSRIs for depression. N Engl J Med. 2006;354:1231–1242.
3. Trivedi MH, Fava M, Wisniewski SR, Thase ME, Quitkin F, Warden D, Ritz L, Nierenberg AA, Lebowitz BD, Biggs MM, Luther JF, Shores-Wilson K, Rush AJ. Medication augmentation after the failure of SSRIs for depression. N Engl J Med. 2006;354:1243–1252.
4. Fava M, Rush AJ, Wisniewski SR, Nierenberg AA, Alpert JE, McGrath PJ, Thase ME, Warden D, Biggs MM, Luther JF, Niederehe G, Ritz L, Trivedi MH. A comparison of mirtazapine and nortriptyline following two consecutive failed medication treatments for depressed outpatients: a STAR*D report. Am J Psychiatry. 2006;163:1161–1172.
5. Nierenberg AA, Fava M, Trivedi MH, Wisniewski SR, Thase ME, McGrath PJ, Alpert JE, Warden D, Luther JF, Niederehe G, Lebowitz BD, Shores-Wilson K, Rush AJ. A comparison of lithium and T3 augmentation following two failed medication treatments for depression: a STAR*D report. Am J Psychiatry. 2006;163:1519–1530.
6. McGrath PJ, Stewart JW, Fava M, Trivedi MH, Wisniewski SR, Nierenberg AA, Thase ME, Davis L, Biggs MM, Shores-Wilson K, Luther JF, Niederehe G, Warden D, Rush AJ. Tranylcypromine versus venlafaxine plus mirtazapine following three failed antidepressant medications trials for depression: a STAR*D report. Am J Psychiatry. 2006;163:1531–1541.
7. Rush AJ, Trivedi MH, Wisniewski SR, Nierenberg AA, Stewart JW, Warden D, Niederehe G, Thase ME, Lavori PW, Lebowitz BD, McGrath PJ, Rosenbaum JF, Sackheim HA, Kupfer DJ, Luther J, Fava M. Acute and longer-term outcomes in depressed outpatients requiring one or several treatment steps: a STAR*D report. Am J Psychiatry. 2006;163:1905–1917.
8. National Institute of Mental Health (NIMH). Sequenced Treatment Alternatives to Relieve Depression (STAR*D) Research Protocol. Washington (DC): NIMH; revised June 28, 2002. Received October 1, 2010, from NIMH following a request under the Freedom of Information Act.
9. Rush AJ, Fava M, Wisniewski SR, Lavori PW, Trivedi MH, Sackeim HA, Thase ME, Nierenberg AA, Quitkin FM, Kashner TM, Kupfer DJ, Rosenbaum JF, Alpert J, Stewart JW, McGrath PJ, Biggs MM, Shores-Wilson K, Lebowitz BD, Ritz L, Niederehe G. Sequenced treatment alternatives to relieve depression (STAR*D): rationale and design. Control Clin Trials. 2004;25:119–142.
10. IsHak WW, Mirocha J, Christensen S, Wu F, Kwock R, Behjat J, Pi S, Akopyan A, Peselow ED, Cohen RM, Elashoff D. Patient-reported outcomes of quality of life, functioning, and depressive symptom severity in major depressive disorder comorbid with panic disorder before and after SSRI treatment in the STAR*D trial. Depress Anxiety. 2014;31:707–716.
11. IsHak WW, Steiner AJ, Klimowicz A, Kauzor K, Dang J, Vanle B, Elzahaby C, Reid M, Sumner L, Danovitch I. Major depression comorbid with medical conditions: analysis of quality of life, functioning, and depressive symptom severity. Psychopharmacol Bull. 2018;48:8–25.
12. Pigott HE, Leventhal AM, Alter GS, Boren JJ. Efficacy and effectiveness of antidepressants: current status of research. Psychother Psychosom. 2010;79:267–279.
13. Pigott HE. STAR*D: a tale and trial of bias. Ethical Hum Psychol Psychiatry. 2011;13:6–28.
14. Pigott, HE. The STAR*D trial: It’s time to reexamine the clinical beliefs which guide the treatment of major depression. Canadian J Psychiatry. 2015;60:9–13.
15. Doshi P, Dickersin K, Healy D, Vedula SS, Jefferson T. Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ. 2013 Jun 13;346:f2865.
16. Kirsch I, Huedo-Medina TB, Pigott HE, Johnson B. Do outcomes of clinical trials resemble those “real world” patients? A re-analysis of STAR*D antidepressant data. Psychology of Consciousness: Theory, Research, and Practice. 2018 https://doi.org/10.1037/cns0000164
17. NIMH Data Archive available at: https://ndar.nih.gov/edit_collection.html?id=2148
Competing interests:
No competing interests
06 March 2019
H. Edmund Pigott
Psychologist
William Dubin (The Lewis Katz School of Medicine, Temple University), Irving Kirsch (Beth Israel Deaconess Medical Center, Harvard Medical School), Jay Amsterdam (Perelman School of Medicine, University of Pennsylvania)
Rapid Response:
Call to action: RIAT reanalysis of the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) Study
We issue a call to action to restore the reporting of STAR*D’s six trials [1-6] and summary article reporting 12-month follow-up findings. [7]
STAR*D (NCT00021528) is the largest and most consequential antidepressant study ever conducted. Funded by the National Institute of Mental Health (NIMH) for 35-million $US, STAR*D enrolled 4,041 patients between 2001-2006 who screened positive for major depression while seeking routine medical or psychiatric care and included patients with common medical and psychiatric comorbid conditions thereby increasing the generalizability of its findings.
STAR*D provided up to 4 treatment steps per patient and was designed to give guidance in selecting the best next-step option for patients who fail to gain sufficient relief from their first, and/or subsequent, trial(s). Antidepressants were administered using a system of measurement-based care that assessed symptoms and side-effects each visit to guide aggressive medication dosing “to ensure that the likelihood of achieving remission was maximized and that those who did not reach remission were truly resistant to the medication.”[1, p. 30]
STAR*D evaluated the relative effectiveness of 11 pharmacologically distinct drug–drug combination treatments in five head-to-head comparisons. [2-6] Patients who achieved remission were encouraged to enter one-year free follow-up care, as were responder patients who failed to attain remission but did not want to continue to the next step as this involved changing medication.
STAR*D’s prespecified primary outcome was remission, defined as scoring <8 on the blindly-administered Hamilton Rating Scale of Depression (HRSD). Response was a secondary outcome defined as ≥50% HRSD reduction. STAR*D had 11 additional prespecified secondary outcome measures assessing health, work and social functioning, quality of life, side-effect burden, medical utilization and cost, and personal income. Both the prespecified primary and secondary outcomes were administered at entry into and exit from each treatment step and follow-up months 3, 6, 9, and 12. [8, table 2]
The investigators state, “STAR*D uses a randomized, controlled design to evaluate both the theoretical principles and clinical beliefs that currently guide the management of treatment-resistant depression in terms of symptoms, function, satisfaction, side-effect burden, and health care utilization and cost estimates. Given the dearth of controlled data, results should have substantial public health and scientific significance.” [9, p. 136]
Thirteen years later, STAR*D researchers have still not published steps 1-4 pre/post and follow-up findings for the secondary outcomes as prespecified. Other researchers though have conducted post-hoc analyses using a subset of the secondary outcomes typically focused on special populations. [eg, 10, 11] Elsewhere we document STAR*D researchers evidenced multiple protocol violations with each violation having an inflationary effect on reported remission and response rates. [12-14]
Given STAR*D’s “substantial public health and scientific significance,” we intend to restore the record for these trials following the principles of the Restoring Invisible and Abandoned Trials (RIAT) initiative. [15]
Our rationale for correcting the record
Apparent protocol violations include:
• Following data collection and without disclosure, substituting a non-blinded/clinic-administered assessment that was used to guide care as the secondary outcome to report remission rates and sole outcome to report response rates in steps 1‐4, and sole outcome measure to report remission and response rates in the summary article. This switch occurred despite STAR*D’s protocol specifically excluding all non-blinded/clinic-administered assessments from use as research outcome measures. [8, p. 47-48]
• Used data from 931 patients deemed ineligible for analysis in step-1 in their steps 2-4 and summary articles without clear disclosure: 607 because these patients’ <14 HRSD score signified at most only mild depressive symptoms when entering the study as well as 324 patients without a blindly-administered baseline HRSD. [1, figure 1] Furthermore, our analysis of STAR*D’s dataset discovered that 99 of the 607 patients scored <8 on their baseline HRSD—STAR*D’s remission criterion. Despite scoring as not depressed at study outset, the outcome data from many of these 99 patients were included in the steps 2-4 articles and all were included in the summary article.
• Failed to disclose that all patients started on citalopram in their baseline visit while excluding from analysis the 370 who then dropped out, [1, figure 1; 7, figure 1] despite the step-1 article stating, “our primary analyses classified patients with missing exit HRSD scores as nonremitters a priori.” [1, p. 34] These 370 early dropout patients did not have an exit HRSD and therefore should have been counted as treatment failures as prespecified.
Recently, we published a step-1 reanalysis using NIMH’s STAR*D dataset. [14] This analysis found substantial reductions in remission and response rates using the blindly-administered HRSD. While step-1 authors claimed “the response and remission rates in this highly generalizable sample with substantial axis I and axis III comorbidity closely resemble those seen in 8-week efficacy trials,” [1, p. 28] our analysis found citalopram’s actual real-world effectiveness was approximately half that reported in efficacy trials.
Surprisingly, an exploratory analysis of step-2’s dataset discovered 117 patients scoring <8 on the HRSD at entry into step-2 yet were included in the step-2 analyses. In other words, 117 patients scored as remitted prior to starting their step-2 medication yet were included in STAR*D’s medication switch [2] and augmentation [3] articles. This occurred despite STAR*D researchers prespecifying, “participants who begin a level with HRSD <8 will be excluded from analyses.” [9, p. 130]
The fact that there were no reported statistical differences in remission and response rates for any of STAR*D’s five head-to-head trials of 11 chemically-distinct medications/medication combinations—and therefore providing no evidence-based guidance for the next best medication option after a failed trial—may be due to these protocol violations versus what actually occurred had the research protocol been followed.
Scope of our restoration
Given the unreported deviations from STAR*D’s protocol, an accurate restoration using the prespecified primary and secondary outcome measures for steps 1-4 and follow-up is essential to understanding the true risks and benefits of each STAR*D treatment, and determine whether to embrace STAR*D’s investigators repeated claims of the superiority of their measurement-based, treat-to-remission, model of care.
Our sources
We will conduct a RIAT reanalysis of the patient-level data in adherence to the original protocol and statistical analysis plan obtained from the NIMH Data Archive unless STAR*D’s authors publicly commit to do so. [17]
Questions our restoration aims to answer
1. What is the fidelity between steps 1-4 published outcomes compared to those from the restored patient-level data using the prespecified outcome measures, inclusion for analysis criteria, and statistical plan?
2. What are the comparative outcomes for steps 2-4 head-to-head trials using the prespecified outcome measures, inclusion criteria, and statistical plan?
3. What are the comparative outcomes during follow-up using the prespecified outcome measures, inclusion criteria, and statistical plan?
References:
1. Trivedi MH, Rush AJ, Wisniewski SR, Nierenberg AA, Warden D, Ritz L, Norquist G, Howland RH, Lebowitz B, McGarth PJ, Shores-Wilson K, Biggs MM, Balasubramani GK, Fava M. Evaluation of outcomes with citalopram for depression using measurement-based care in STAR*D: implications for clinical practice. Am J Psychiatry. 2006;163:28–40.
2. Rush AJ, Trivedi MH, Wisniewski SR, Stewart JW, Nierenberg AA, Thase ME, Ritz L, Biggs MM, Warden D, Luther JF, Shores-Wilson K, Niederehe G, Fava M. Bupropion-SR, sertraline, or venlafaxine-XR after failure of SSRIs for depression. N Engl J Med. 2006;354:1231–1242.
3. Trivedi MH, Fava M, Wisniewski SR, Thase ME, Quitkin F, Warden D, Ritz L, Nierenberg AA, Lebowitz BD, Biggs MM, Luther JF, Shores-Wilson K, Rush AJ. Medication augmentation after the failure of SSRIs for depression. N Engl J Med. 2006;354:1243–1252.
4. Fava M, Rush AJ, Wisniewski SR, Nierenberg AA, Alpert JE, McGrath PJ, Thase ME, Warden D, Biggs MM, Luther JF, Niederehe G, Ritz L, Trivedi MH. A comparison of mirtazapine and nortriptyline following two consecutive failed medication treatments for depressed outpatients: a STAR*D report. Am J Psychiatry. 2006;163:1161–1172.
5. Nierenberg AA, Fava M, Trivedi MH, Wisniewski SR, Thase ME, McGrath PJ, Alpert JE, Warden D, Luther JF, Niederehe G, Lebowitz BD, Shores-Wilson K, Rush AJ. A comparison of lithium and T3 augmentation following two failed medication treatments for depression: a STAR*D report. Am J Psychiatry. 2006;163:1519–1530.
6. McGrath PJ, Stewart JW, Fava M, Trivedi MH, Wisniewski SR, Nierenberg AA, Thase ME, Davis L, Biggs MM, Shores-Wilson K, Luther JF, Niederehe G, Warden D, Rush AJ. Tranylcypromine versus venlafaxine plus mirtazapine following three failed antidepressant medications trials for depression: a STAR*D report. Am J Psychiatry. 2006;163:1531–1541.
7. Rush AJ, Trivedi MH, Wisniewski SR, Nierenberg AA, Stewart JW, Warden D, Niederehe G, Thase ME, Lavori PW, Lebowitz BD, McGrath PJ, Rosenbaum JF, Sackheim HA, Kupfer DJ, Luther J, Fava M. Acute and longer-term outcomes in depressed outpatients requiring one or several treatment steps: a STAR*D report. Am J Psychiatry. 2006;163:1905–1917.
8. National Institute of Mental Health (NIMH). Sequenced Treatment Alternatives to Relieve Depression (STAR*D) Research Protocol. Washington (DC): NIMH; revised June 28, 2002. Received October 1, 2010, from NIMH following a request under the Freedom of Information Act.
9. Rush AJ, Fava M, Wisniewski SR, Lavori PW, Trivedi MH, Sackeim HA, Thase ME, Nierenberg AA, Quitkin FM, Kashner TM, Kupfer DJ, Rosenbaum JF, Alpert J, Stewart JW, McGrath PJ, Biggs MM, Shores-Wilson K, Lebowitz BD, Ritz L, Niederehe G. Sequenced treatment alternatives to relieve depression (STAR*D): rationale and design. Control Clin Trials. 2004;25:119–142.
10. IsHak WW, Mirocha J, Christensen S, Wu F, Kwock R, Behjat J, Pi S, Akopyan A, Peselow ED, Cohen RM, Elashoff D. Patient-reported outcomes of quality of life, functioning, and depressive symptom severity in major depressive disorder comorbid with panic disorder before and after SSRI treatment in the STAR*D trial. Depress Anxiety. 2014;31:707–716.
11. IsHak WW, Steiner AJ, Klimowicz A, Kauzor K, Dang J, Vanle B, Elzahaby C, Reid M, Sumner L, Danovitch I. Major depression comorbid with medical conditions: analysis of quality of life, functioning, and depressive symptom severity. Psychopharmacol Bull. 2018;48:8–25.
12. Pigott HE, Leventhal AM, Alter GS, Boren JJ. Efficacy and effectiveness of antidepressants: current status of research. Psychother Psychosom. 2010;79:267–279.
13. Pigott HE. STAR*D: a tale and trial of bias. Ethical Hum Psychol Psychiatry. 2011;13:6–28.
14. Pigott, HE. The STAR*D trial: It’s time to reexamine the clinical beliefs which guide the treatment of major depression. Canadian J Psychiatry. 2015;60:9–13.
15. Doshi P, Dickersin K, Healy D, Vedula SS, Jefferson T. Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ. 2013 Jun 13;346:f2865.
16. Kirsch I, Huedo-Medina TB, Pigott HE, Johnson B. Do outcomes of clinical trials resemble those “real world” patients? A re-analysis of STAR*D antidepressant data. Psychology of Consciousness: Theory, Research, and Practice. 2018 https://doi.org/10.1037/cns0000164
17. NIMH Data Archive available at: https://ndar.nih.gov/edit_collection.html?id=2148
Competing interests: No competing interests