Commentary: Measuring what matters: the case for patient generated PROMSBMJ 2015; 350 doi: https://doi.org/10.1136/bmj.h54 (Published 10 February 2015) Cite this as: BMJ 2015;350:h54
When I started researching amyotrophic lateral sclerosis (ALS) in 2002 I was trained to interview patients and administer a 12 item scale assessing their ability to walk, speak, and breathe. It yielded a score between 48 (healthy) and 0 (incapacitated). This functional rating scale (ALSFRS-R)1 was developed to support clinical trials and correlates with survival and markers of neurodegeneration.2 3 In contrast to the shared notes schemes Nelson and colleagues’ describe,4 we were not allowed to tell patients what their score was. The scale was seen as a research tool to describe groups, not the progress of individuals, so would it help or harm patients to know they scored 23 or 35?
Subsequent validation studies showed the measure was just as reliable if administered by caregivers,5 by telephone,6 or by patients themselves.7 In November 2005 patients were given access to an online self reported version of the scale and visualisations of their progress compared with that of other patients on PatientsLikeMe.com. This website was built by a family affected by ALS who wanted to learn more about the progression of the disease and manage it effectively. Self reported data submitted to the website were validated against clinical measures8 and used to help understand the effect of treatments.9 Crowdsourced models of ALSFRS-R scores recorded by clinicians have been used to predict outcomes, and patient reported scores might achieve the same goal. Tellingly, the predictive algorithms designed by mathematically minded solvers who were non-experts in the disease substantially outperformed the predictions of ALS expert clinicians.10
Regrettably, few other groups of patients have such well developed patient reported outcome measures (PROMs) for their particular conditions. Done well, such tools can serve so many purposes: identifying treatment effects, mapping to pathology, supporting predictive modelling for researchers, guiding clinical care, and offering feedback to support self management. In the absence of a common framework to systematically evaluate PROMs against these purposes many who develop them continue to produce tools that fulfil only a subset of these ambitions. Part of the problem is that clinicians often lack psychometric expertise or the time and funding to develop more useful tools. As a result, many PROMs are sponsored by the pharmaceutical industry, in line with Food and Drug Administration guidelines,11 to support label claims for the added value of their medicines to improve symptom relief or quality of life. Some claim that an unintended consequence of this has been the development of a “cottage industry” of companies developing measures with the needs of pharmaceutical sponsors primarily in mind.12 Furthermore the sharing of such tools may be restricted through licensing because the measurements are considered to offer a competitive advantage.
Alternative paths to systematically developing PROMs are, however, being explored. The PROMIS initiative, a group of academic researchers, develops measures that use computerised adaptive testing (CAT).13 Unlike traditional measures, which use a fixed list of items, CAT adjusts the level of questions according to the patient’s previous answers. CAT shows good performance against traditional measures,14 but understanding responses requires digital administration and new skills, and, crucially, no PROMIS instrument has yet been used as an endpoint to support an FDA label claim.12
From the patient side, “patient powered research networks” such as PCORnet (www.pcornet.org/) offer the opportunity for large online registries of patients to contribute to the psychometric development of new PROMs.15 On another network, PatientsLikeMe, the Open Research Exchange (www.openresearchexchange.com) lets researchers rapidly create and validate measures with substantial patient input16 by combining an engaged patient network with online software that guides researchers through the process of developing PROMs with constant patient feedback. Instruments developed on the Open Research Exchange are licensed under Creative Commons, meaning they are free for use and adaptation, and work is under way to understand how PROMs developed online might meet FDA guidelines in a reliable and resource effective manner.17
Measures that are more patient centred might frame questions in a more positive and aspirational manner, use the patient’s own language, and harness CAT to be brief and responsive, automatically honing in on relevant domains across comorbidities to reduce the survey burden. Patients using technology to take the lead in methods for measuring their diseases could be a cornerstone of the learning health system—tools developed by patients for patients that align what’s measured to what matters (box). Measures optimising the priorities of patients, clinicians, and researchers that are then overlaid on patient portals that integrate with clinical practice18 could improve clinical care19 and self management and fuel better predictive modelling, pragmatic trials, n of 1 trials, and comparative effectiveness research. Now is the time to measure what matters.
Patients codeveloping PROMs
Cathy Wolf worked as a psychologist at IBM for 25 years before ALS prevented her from working. Although she is quadriplegic and dependent on a ventilator, her intellect remains. When she came to complete the ALSFRS-R she was frustrated that despite her ability to participate in family life and write poetry, the scale reflected her as “a zero.” She uses an eye tracking machine and a computer to communicate, but the scale rated her communication as zero because she cannot speak or hold a pen. In response, Cathy worked with PatientsLikeMe to test and validate new items with input from over 300 patients, many of whom would be too impaired to take part in traditional research but could do so thanks to the internet and assistive technologies. The ALSFRS Extension20 has now been translated into other languages21 and is used in research, including the Department of Veterans Affairs Biorepository Brain Bank.22 Coproduction of PROMs with patients could bring new perspectives and illuminate blind spots—it’s also the publication of which Cathy is proudest.23
Cite this as: BMJ 2015;350:h54
Competing interests: PW is an employee of PatientsLikeMe, which receives funding from the pharmaceutical industry for research including the development of patient reported outcome measures. PW is principal investigator on a grant awarded by the Robert Wood Johnson Foundation for the development of the Open Research Exchange (ORE). Further details of PW’s disclosures are available on thebmj.com.