Outcome selection and role of patient reported outcomes in contemporary cardiovascular trials: systematic reviewBMJ 2010; 341 doi: http://dx.doi.org/10.1136/bmj.c5707 (Published 01 November 2010) Cite this as: BMJ 2010;341:c5707
- Kazem Rahimi, specialist registrar and senior clinical researcher1,
- Aneil Malhotra, specialist registrar in cardiology1,
- Adrian P Banning, consultant cardiologist1,
- Crispin Jenkinson, professor of health services research2
- 1Department of Cardiology, Oxford Heart Centre, John Radcliffe Hospital, Oxford OX3 9DU, UK
- 2Department of Public Health, Health Services Research Unit, University of Oxford
- Correspondence to: K Rahimi
- Accepted 25 August 2010
Objectives To systematically assess the type of outcomes selected and the prevalence of patient reported outcomes in contemporary cardiovascular trials and to quantify any misuse or underuse of patient reported outcomes using a specially developed tool that would allow estimation of the relevance of such outcomes to clinical decision making.
Design Systematic review.
Data sources Medline and Embase.
Study selection Randomised controlled trials of the treatment for or prevention of cardiovascular disease published in 10 leading general medical and cardiology journals from January 2005 to December 2008.
Results Primary outcomes were patient important (death, morbidity, or patient reported outcomes) in only 93 of 413 trials (23%, SE 2%), whereas another 92 (22%, SE 2%) combined these outcomes with other less important ones into a composite. Sixty five trials (16%; SE 2%) used at least one instrument to measure patient reported outcomes, mostly in trials where such information would have been important or crucial for clinical decision making (52 trials). Patient reported outcomes were judged to be of little incremental value to a large number of, mostly explanatory, cardiovascular trials (152 trials). However, many trials in which patient reported outcomes would have been important or crucial for clinical decision making did not report such outcomes (122 of 174 trials, 70%). These included several trials that primarily aimed to improve symptoms or functional status, trials that tested interventions with a considerable potential for causing harm (mainly bleeding) that were not meaningfully measured, and trials with composite outcomes that were dominated by outcomes of questionable importance to patients.
Conclusions Despite a continued rise in the reporting of patient reported outcomes with no evidence for their misuse in more recent cardiovascular trials, they seem to be still underused once their relevance to clinical decision making has been taken into account. This was largely explained by inappropriate use of composite outcomes and inadequate measurement of harms.
The ultimate goal of clinical trials is to identify interventions to benefit future patients. The selection of best variables for measuring such benefits or harms is complex, however, and has been open to debate.1 2 Perceived from the patients’ perspective, medical interventions should either prevent premature death or improve health related quality of life. Instruments for patient reported outcomes allow measurement of the effect of interventions from the patients’ perspective.3 Such instruments can be particularly useful when interventions aim to improve symptoms and functional status (for example, the walking impairment questionnaire4) or health related quality of life in a wider sense (for example the chronic heart failure questionnaire5).
Earlier studies have suggested that patient reported outcomes might have been underused in randomised trials in general6 7and more so in cardiovascular trials.6 These studies were, however, published several years ago when suitable instruments for measuring patient reported outcomes were limited and only a few researchers had the necessary expertise for using them. More importantly, no information is available on the relevance of the use of patient reported outcomes to contemporary cardiovascular trials. It may be that such instruments are less relevant to many cardiovascular trials, where traditionally the focus has been on major clinical outcomes such as myocardial infarction and stroke rather than the relief of symptoms. Furthermore, many cardiovascular trials are concerned with biological effects of interventions, without any intentions on changing clinical practice. For example, when investigators seek to understand the effect of high dose folic acid on coronary vasodilation8 or to explore the pharmacodynamics and pharmacokinetics of different dosages of an antiplatelet agent,9 the measurement of patient reported outcomes is expectedly of little value. Therefore, without assessment of the relevance of patient reported outcomes to individual trials and their impact on clinical decision making, no reliable conclusions can be made about their misuse or underuse.
We assessed the types of outcomes selected and the prevalence of patient reported outcomes in recent cardiovascular trials. To quantify any current misuse or underuse of patient reported outcomes we further sought to develop a tool that would allow estimation of the relevance of such outcomes to clinical decision making and applied it to a set of cardiovascular trials.
We searched Medline and Embase for articles with any of the subject terms “cardiovascular diseases” or “cardiovascular agents” published between January 2005 and December 2008 in any of the following leading general medical or cardiology journals: New England Journal of Medicine, Lancet, JAMA, Annals of Internal Medicine, PLoS Medicine, BMJ, Circulation, Journal of the American College of Cardiology, Circulation Research, and European Heart Journal (see web extra appendix 1 for the detailed search strategy).
Review methods and data extraction
Figure 1⇓ summarises the retrieval and selection process for studies. The combined searches yielded 2374 reports, of which 1376 were excluded on the basis of titles and abstracts. The remaining 998 reports were allocated a unique random number with the use of a random sequence generator.10 We assessed in full text the first 500 articles according to this random list for their eligiblity, and extracted relevant data from eligible trials. Of these, 87 did not meet the eligibility criteria. We used an electronic data extraction form to capture pivotal information from each trial. We categorised reported outcomes and individual components of any composite outcomes into patient important outcomes (death, major morbidity, patient reported outcome) or intermediate outcomes—that is, those not immediately or directly important to patients, such as biochemical, physiological, or surrogate outcomes. The categorisation of outcome was based on previously published ranking systems,11 12 with more emphasis on the methods of outcome measurement (see also web extra appendix 2). From the trials evaluated, we assessed those that were reporting patient reported outcomes to see whether a previously validated instrument was used or whether psychometric properties were reported or referenced in the article. All data were extracted in duplicate. The results and disagreements were discussed until consensus was reached.
Development and validation of a tool for ranking relevance of patient reported outcomes
Several articles described seven pivotal features of trials that would need to be considered when making decisions about the appropriateness of patient reported outcomes in clinical trials.3 7 13 14 15 16 17 18 19 20 We used the proposed framework from these studies to devise a tool that would allow a systematic and reliable ranking of the relevance of patient reported outcomes in cardiovascular trials into five categories (crucial, important, potentially relevant, irrelevant, and uncertain) by the level of importance of patient reported outcomes to clinical decision making. With the lack of any gold standards, we sought expert opinion to reach consensus about the validity of the tool. An initial version of the tool, together with a set of instructions for its use, was sent to nine experts who were identified on the basis of previous publications in the area of outcome measurement and health services research. After incorporation of the comments received from four experts, two reviewers (KR and AM) independently applied the modified and face validated tool to 100 randomly selected articles. The experience gained from the initial data extraction was used to inform the final tool (fig 2⇓).
Web extra appendix 2 provides a detailed description of the tool. Briefly, a patient reported outcome was considered irrelevant in studies that primarily sought to test a hypothesis useful for understanding the differences between intervention strategies, without any claims on changing clinical practice. Studies that aimed to help make decisions about alternative strategies were categorised as pragmatic trials. If the main goal of such trials was to improve symptoms, functional status, or quality of life, we considered the patient reported outcome to be crucial. Otherwise, ranking of the relevance of patient reported outcomes depended on several other trial features (see fig 2 and web extra appendix 2).
Prevalence of patient reported outcomes over time
One study assessed the prevalence of quality of life in cardiovascular trials and showed that between 1980 and 1997, 2.1% of such trials reported quality of life and that the rate increased over time (from 0.34% in 1980 to 3.6% in 1997).6 We replicated and updated that reported search and extended the timeframe to 2009 to assess the effect of time on reporting of quality of life in cardiovascular trials.
We used descriptive statistics to characterise the included trials. Univariate analyses (χ2 test, Fisher’s exact test, and Mantel-Haenszel test for trend) explored associations of trial characteristics and type of outcome (patient important: yes or no) and measurement of patient reported outcomes. In the first 50 reviewed trials the prevalence for use of patient reported outcomes was 14%. We therefore estimated that data from about 400 trials would provide sufficient information for assessment of associations between trial features and type of outcomes selected. To allow for several ineligible trials and duplicate reports, we set a target of 500 articles to be reviewed in full text.
Excluding the first 100 trials that were evaluated in the pilot phase for development of the ranking tool, we used information from the next set of randomly selected trials, independently extracted by two reviewers (KR and AM), to estimate the chance corrected inter-rater reliability к. We weighted the outcomes to account for the closeness of different categories, as shown in the matrix (see web extra appendix 3). Assessment of inter-rater reliability based on the final 330 reports showed a substantial chance corrected inter-rater agreement between the two investigators: weighted к=0.79; quadratic weighted к=0.77. All analyses were carried out using STATA v 9.1.
Type of reported outcomes
In 93 trials (23%, SE 2%) the primary outcome was patient important (death, major morbidity, patient reported outcome), whereas another 92 trials (22%, SE 2%) combined these with intermediate outcomes of less or no importance to patients (table 1⇓). In 52 of these 92 trials (57%) less than half of the events were attributable to patient important outcomes—that is, the composite outcome was dominated by intermediate outcomes such as physiological, biochemical, or surrogates. All other trials (228 (55%), SE 2%) used intermediate outcomes as their primary outcome. Patient reported outcomes were reported in 65 of the 413 trials assessed (16%, SE 2%) but were the primary outcome in only two trials and a coprimary outcome in another four trials.
Table 1 shows the association between trial characteristics, the type of reported outcomes, and the presence or absence of patient reported outcomes. Contrasting the significant association between the type of primary outcome and several features of the trials, there was no evidence that the trials reporting patient reported outcomes were significantly different from those not reporting patient reported outcomes, based on the type of primary end point selected, year of publication, trial design, blinding methods, number of participants, number of centres, study coordinating region, or the type of funding. Strong evidence was found that studies reporting patient reported outcomes differed by the condition under investigation and the type of intervention. In particular, studies that included patients with heart failure, arrhythmia (for example, atrial fibrillation) or “other vascular conditions” (for example, pulmonary arterial hypertension, peripheral arterial disease, stroke) were more likely to use patient reported outcomes than those that included mainly people with coronary artery disease or when interventions were aimed at primary prevention of cardiovascular conditions. Studies that assessed the effect of lifestyle interventions, nutrition, and those with “other interventions,” such as reminder systems or screening tests, were also more likely to use patient reported outcomes, whereas studies of coronary intervention or other procedures were less likely to do so.
Quality and types of reported instruments for measuring patient reported outcomes
Of the 65 trials reporting patient reported outcomes, only seven (11%) used ad hoc measures (that is, no previously validated instrument used or psychometric properties reported or referenced) or proxies for assessment of patients’ health status, and in another three (5%) published information was insufficient to ascertain the properties of the instruments used. The other 55 trials (85%) used at least one validated instrument (seven trials reported three different instruments and 16 reported two instruments). The most commonly used disease specific instruments were the Minnesota living with heart failure questionnaire and the Seattle angina questionnaire, which were used in 18 and five different studies, respectively. The most commonly used generic instruments were the short-form 36 health survey and the EuroQol 5-D, which were used in nine and five studies, respectively.
Prevalence of reporting on patient reported outcomes over time
The replicated and updated search from a previous study showed that the proportion of cardiovascular trials that also mentioned quality of life has increased more than threefold since 1997 (fig 3⇓). By 2009, about 14% of trials mentioned quality of life. However, these crude figures should be interpreted with caution as they were only based on a simple bibliographic search, which may not reflect actual use of patient reported outcomes.
Relevance of reporting on patient reported outcomes
Table 2⇓ shows the assessed level of relevance of patient reported outcomes overall and by the status of reporting of such outcomes to individual trials. Patient reported outcomes were found to be crucial for informed clinical decision making in 93 trials (23%). These were mostly (n=57) trials that had set out chiefly to improve patients’ wellbeing or quality of life. For example, the Canadian Cardiac Randomized Evaluation of Antidepressant and Psychotherapy Efficacy (CREATE) trial investigated the effect of citalopram and interpersonal psychotherapy on depression in people with coronary artery disease (and had used two validated scoring tools for the assessment of depression).21 A few other studies in the category of “patient reported outcome crucial” were trials that aimed to change clinical practice, but their outcomes were dominated by ones of questionable importance to the patients. At the same time these trials failed to measure adequately the effect of harms or had included patients who had chronic symptoms for whom improvement in quality of life would have been highly desirable. For example, the Plasma Brain natriuretic Peptide-Guided Therapy to Improve Outcome in Heart Failure: The STARS-BNP Multicenter Study trial assessed the effect of plasma brain natriuretic peptide guided therapy on death and admission to hospital in patients with chronic heart failure.22 However, deaths due to heart failure accounted for only about 15% of the composite outcome, with the remainder being admissions to hospital. Overall, 37 of the 93 trials in which patient reported outcomes would have been crucial reported such outcomes.
Patient reported outcomes were judged to be important in 81 of the trials (20%), of which 15 reported such outcomes. The majority of these were superiority trials that had used mostly patient important outcomes but had not adequately measured harms or had included a patient population with chronic symptoms. Studies that did not measure harm adequately were mostly trials of interventions known to be associated with a substantial bleeding risk. For example, the value of a decrease in blood haemoglobin levels, which was commonly used as a criterion for major bleeding, is not obvious from the patient’s perspective. The combination of such harms with potential benefits of the intervention has been used in an attempt to estimate the net clinical benefit.23 However, such combined effects without any valuation by the patients have made the interpretation of the results even more challenging.24
In another 59 trials (14%) patient reported outcomes were judged to be potentially relevant. These were trials that aimed to inform clinical decision making beyond the relief of symptoms or improvements in quality of life, had mainly used patient important outcomes as the primary end point, and had measured harm adequately (if expected or possible). In this setting, patient reported outcomes were thought to be only potentially relevant, mainly for estimation of the preference based outcomes of the intervention and the calculation of quality adjusted life years as part of economic evaluation.
Patient reported outcomes were judged to be irrelevant in 93 trials (23%). These were explanatory trials that did not intend to inform clinical decision making directly. For example, one trial investigated the effect of angiotensin receptor blockers on blood kinin levels to explore possible mechanisms of action of this type of drugs.25 Only two trials in this category used patient reported outcomes, giving little evidence for the possibility of misuse of patient reported outcomes in circumstances where such information would be of limited value.
The relevance of patient reported outcomes was ranked as uncertain in 87 trials (21%). These trials were primarily limited by the lack of patient important outcomes to address a clinically relevant question. In such trials, judgment on the incremental value of patient reported outcomes would have been highly hypothetical. For example, in a trial that aimed to assess the efficacy of reperfusion therapy 12 hours after the onset of symptoms in patients with acute myocardial infarction but used infarct size as its primary end point,26 assessment of the level of relevance of patient reported outcomes would have been secondary because to make an informed decision on patient care such a trial would in the first instance be expected to use patient important outcomes before the incremental value of patient reported outcomes could be evaluated.
Patient reported outcomes were reported in 16% of contemporary cardiovascular trials published in major cardiology and general medical journals during 2005-8. Reported patient reported outcomes were mostly validated instruments and mostly used in settings where such information would have been highly relevant to clinical decision making. Although patient reported outcomes were judged to be of little value to a large proportion of, mostly explanatory, cardiovascular trials, still more than two thirds of trials in which patient reported outcomes were judged to be important or crucial for clinical decision making failed to report such outcomes. In addition, composite outcomes in many trials were often dominated by outcomes that were not immediately important to patients.
Three major areas were identified in this study where the measurement of patient reported outcomes was underused but could have made a valuable contribution to interpretation of the study findings. Firstly, many trials that were primarily concerned with relief of symptoms and improvement in quality of life failed to use patient reported outcomes, and even when they did these were rarely the primary outcome (contributing to the lack of impact of such findings on informing clinical practice27).
Secondly, shortcomings in reporting of patient reported outcomes were also observed in trials that assessed benefits of an intervention in light of considerable harms caused by the same intervention. While it could be argued that most interventions are likely to have multiple effects, desired and undesired, expected and unexpected, it would be inappropriate to demand measurement of patient reported outcomes in all trials. Standard safety monitoring of trials (despite all its limitations) and crude reporting of events might be sufficient for clinical decision making in circumstances where the spectrum of harms is reasonably well known and clinically measurable.28 However, issues arise when potential harms are substantial and their objective evaluation is not directly relevant to patients. One common area identified in this study was the assessment of the risk of bleeding in trials aiming to reduce the risk of cardiovascular events. The investigators used a diversity of methods for measurement and classification of bleeding, with some of these being purely based on objective laboratory measures with little relevance to patients. This bears the risk of invalidating comparisons between the different outcome components within the same study, as well as between studies reporting similar outcomes.29 A few investigators have attempted to estimate the net clinical effect by combining harms and benefits into a single composite outcome. However, such an approach would assume that detection methods for different types of outcomes are equally sensitive and that different types of events have a similar clinical weight. Since such conditions cannot easily be met in practice,30 and in the absence of valid and meaningful scoring systems for measurement of such harms, patient reported outcomes could offer an alternative method for assessment of the impact of such effects on patients. Preference based instruments for patient reported outcomes (that is, instruments that provide relative values for different health states23), would allow an estimate of the weighted average of benefits and harms from the patient’s perspective and could provide a common denominator for different types of outcomes.
Thirdly, the use of patient reported outcomes could be improved in the group of trials using composite end points that incorporate outcomes of different importance to the patients. This can be particularly problematic when outcomes that are less important to the patients dominate the results or when treatment effects are discordant and qualitatively different across the individual components. The current study shows that in over half of the studies with mixed primary outcomes, the findings were dominated by the intermediate outcomes of questionable relevance to patients. A generic preference based instrument for patient reported outcomes would have the ability to evaluate the overall effect of such outcomes from the patient’s perspective.31 Serial measurements would also circumvent problems with time based analyses that commonly count the first occurrence of events only and ignore recurrences, even if these are of greater importance to patients.
Concerns have been raised that widespread recommendations from funding agencies and regulatory bodies for the use of patient reported outcomes in clinical trials may lead to inappropriate use of such outcomes. In this study only 5% of trials in which patient reported outcomes were ranked as irrelevant reported such outcomes, therefore dismissing any concerns for substantial misuse. However, this may be an underestimate because our protocol included high quality medical journals only and the trials may not be representative of all cardiovascular trials. In addition, the prevalence of reporting patient reported outcomes was based on the published information. Some investigators may have measured but not reported such outcomes.27
Strengths and limitations of the review
To the best of our knowledge, this is the first study that has devised a tool for the reliable and transparent assessment of the relevance of patient reported outcomes to cardiovascular trials. The tool was validated by experts, applied to a large set of cardiovascular trials with a high inter-rater reliability, and may guide researchers in the future at the design stage of trials. Although a certain degree of judgment was inevitable in categorising trials, discrepancies between independent assessors were small and easily resolved through discussion, with no material effect on the overall findings. The focus of the current study was the assessment of patient reported outcomes as study outcome measures. We acknowledge that patient reported outcomes can play an important part at other stages of clinical trials, such as during recruitment to identify potentially eligible participants or at baseline for categorisation of participants into different risk groups.32 We also acknowledge that patient reported outcomes will not be the panacea for all existing challenges of outcome selection in clinical trials. Lack of suitable instruments and the somewhat sophisticated requirements for analysis and interpretation of results33 may have contributed to their underuse. Inadequate blinding of study participants may also prevent investigators from using patient reported outcomes, as such outcomes are more prone to bias than more objective outcomes. However, we found no evidence that the type of blinding correlated with use of patient reported outcomes. Conduct of trials is time consuming and many of the assessed trials may have been initiated a few years before publication, when guidelines on use of patient reported outcomes in trials were unavailable. Other feasibility issues such as the need for larger sample sizes and longer duration of follow-up may also be difficult to overcome. However, the study size or number of participants was not associated with the use of patient reported outcomes in the current study. This study was not designed to investigate the underlying causes of underuse of patient reported outcomes. Qualitative studies may be useful to help understand the investigators’ approach towards outcome selection and to explain why selected outcomes are often inadequate.
Conclusions and policy implications
Despite the continued rise in reporting of patient reported outcomes in cardiovascular trials, such outcomes were still underused in many trials, even after taking account of their relevance to individual trials. The present study reiterates the need for selection of more patient focused outcomes and highlights some of the problems associated with composite outcomes and assessment of harms in cardiovascular trials. In such areas, more widespread use of patient reported outcomes may be a valuable complement to the traditional clinical outcomes. Involvement of patients and experts in outcomes at the design stage of clinical trials may help in the process of selection of outcomes that are relevant and meaningful to those who potentially will be affected by them.
What is already known on this topic
Patient reported outcomes assess the effect of trial interventions from patients’ perspectives and are particularly useful when interventions aim to improve symptoms or functional status
Studies report a low prevalence of patient reported outcomes particularly in cardiovascular trials, perhaps because many cardiovascular interventions aim to affect mortality and major morbidity
Little information is available on the appropriateness of the use of patient reported outcomes in contemporary cardiovascular trials
What this study adds
The prevalence of reporting patient reported outcomes in cardiovascular trials published in high quality medical and cardiology journals has increased to 16%, with no evidence of misuse
About two thirds of trials in which patient reported outcomes were judged to be important or crucial for clinical decision making still failed to report such outcomes
This was largely due to inappropriate use of composite outcomes and inadequate measurement of harms
Cite this as: BMJ 2010;341:c5707
We thank the students and lecturers at the London School of Hygiene and Tropical Medicine and colleagues in Oxford for the inspiring discussions that helped shape the research question, and Stuart Pocock, Peter Fayers, Jonathan Emberson, and Andrew Hutchings for their thoughtful comments on the tool for the assessment of the relevance of patient reported outcomes or on the earlier versions of the manuscript.
Contributors: All authors contributed to the design of the study, data interpretation, and critical revision of the article including its final approval. In addition, KR analysed the data analysis and is guarantor of the study.
Funding: This study received no funding.
Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that (1) none has received financial support for the submitted work; (2) none has any relationships with companies that might have an interest in the submitted work in the previous 3 years; (3) none of their spouses, partners, or children has any financial relationships that may be relevant to the submitted work; and (4) none has any non-financial interests that may be relevant to the submitted work.
Data sharing: The full dataset is available from the corresponding author at.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.