Reporting harms more transparently in trials of cancer drugsBMJ 2018; 363 doi: https://doi.org/10.1136/bmj.k4383 (Published 01 November 2018) Cite this as: BMJ 2018;363:k4383
- Bishal Gyawali, postgraduate trainee1,
- Tomoya Shimokata, assistant professor1,
- Kazunori Honda, medical oncologist2,
- Yuichi Ando, professor1
- 1Department of Clinical Oncology and Chemotherapy, Nagoya University Hospital, Japan
- 2Department of Clinical Oncology, Aichi Cancer Center Hospital, Nagoya, Japan
- Correspondence to: B Gyawali
The clinical trial report of ribociclib, a drug for breast cancer, mentions in its discussion that “Most patients had an acceptable adverse-event profile.”1 A report of a trial of liposomal irinotecan in pancreatic cancer states in the concluding paragraph that it “has a manageable and mostly reversible safety profile.”2 And a trial of tasquinimod in patients with prostate cancer reports “the tolerability was good overall.”3
All three of these studies were published in top medical journals. Naturally, readers would take these statements to be true. However, a look at the data for adverse events doesn’t paint as good a picture. In the first study, more than twice as many patients in the ribociclib arm as in the control arm experienced severe (grade 3 or higher) adverse events (271/334 v 108/330).1 The difference in treatment related serious adverse events (leading to death, life threatening condition, hospital admission or prolonged admission, disability or permanent damage, congenital anomaly or birth defect, or that required medical or surgical intervention to prevent one of the other outcomes4) was nearly five times higher (25 v 5). The trial of liposomal irinotecan that mentioned “manageable and mostly reversible” toxicities in fact shows that five patients in the intervention arm died from drug toxicities versus none in the control.2 In the trial reporting overall good tolerability of tasquinimod, the incidences of severe and serious adverse events compared with control were 42.8% v 33.6% and 36.0% v 23.6%, respectively.3
These three studies are only representative examples. The adverse event profiles of many new cancer drugs are hidden behind similarly general or subjective terms that obscure their harms. We therefore investigated how often publications of cancer drug trials downplayed harms. Based on our experience with reading trial publications, we defined downplaying as use of the following terms or their derivatives to describe adverse events: tolerable, favourable, acceptable, manageable, feasible, and safe. Box 1 explains why their use is inappropriate, irrespective of whether the toxicities were increased or decreased.
Terms used to downplay the harms of cancer drugs and reasons for avoiding them
Acceptable—Acceptable to whom? Were the patients asked if the toxicities were acceptable to them?
Manageable—Serious events and deaths can never be considered manageable. Even manageable toxicities incur burden and decrease patients’ quality of life
Feasible—What is the threshold for feasibility of a treatment? Will the mention of “the treatment is feasible” be enough to obtain patient’s consent to a treatment?
Favourable toxicity profile—Favourable compared with what? Threshold of enduring toxicities and thus favourability is different from patient to patient
Tolerable or well tolerated—Only the patient can decide whether any side effect is tolerable
Safe—Any cancer treatment that has resulted in a treatment related death cannot be considered safe
We examined all phase II or III randomised trials published during 2016 in the five major medical journals, based on their impact factors, that publish cancer drug trials (New England Journal of Medicine, Lancet, Lancet Oncology, Journal of the American Medical Association, and Journal of Clinical Oncology). These five journals capture most randomised trials of new cancer drugs, and almost all trials of new cancer drugs that get approved and make it to the market. We chose trials published in 2016 as it was the most recent calendar year (this research was conducted in 2017). We looked for the identified terms and any others that could imply downplaying of harms. Any dispute, or the discovery of any new term that seemed to downplay the toxicities, would be resolved by discussion and consensus among the authors.
We then assessed how harms in the experimental arm were reported. We extracted the data on severe and serious adverse events and deaths for both the experimental and control cohorts from these trials. All the study eligibility confirmations and data extractions were done twice—once by BG and once by KH, who remained blinded to each other’s data—and finally double checked by TS.
Description of harms
We identified a total of 122 trials of cancer drugs in the journals, of which 53 (43%) contained terms that downplayed harms. Fourteen of the 53 studies did not report any data on severe adverse events, 22 had no data on serious events, and two had no data on deaths. Such under-reporting of harms is common in oncology trials.56 However, when trials mention an acceptable, tolerable, or favourable toxicity profile in the experimental treatment arm, it seems wrong not to report the supporting data.
In the trials that did report data, the rates of severe adverse events were higher in the experimental arm than in the control arm for 77% of trials (30/39), serious adverse events were higher in 84% of trials (26/31), and deaths in 66% of trials (34/51). Thus, despite using terms such as favourable and tolerable to describe the harms profiles of new treatments, the trials often showed a greater number of harms than in the control arms.
Why is transparency important?
Not fully reporting the harms of cancer drugs is of particular concern because cancer drugs usually provide modest benefits at high costs—in terms of both price and toxicities.7 Downplaying harms can suggest a better risk-benefit profile than actually exists.
Describing harms as acceptable or tolerable in trials is unacceptable, irrespective of incidence and risk, as it makes a subjective judgment. Whether harms are acceptable is for individual patients to decide rather than physicians or trial stakeholders, and the threshold for tolerability to harms will differ from person to person. Without collecting data from patients on what they would acknowledge as acceptable or tolerable toxicities, we believe that investigators cannot put those labels on the experiences of our patients. Furthermore, any cancer drug that has ever had a treatment related death shouldn’t be described as safe or as having “manageable toxicities.”
We don’t intend to promote or discourage a certain drug as safe or unsafe. Indeed, one trial cannot provide enough data on safety; ongoing real world data as well as physicians’ and patients’ experience with the use of a drug should guide discussions of toxicity in clinical practice. However, unambiguous and complete reporting of harms data in trial publications is an important step to appropriate clinical practice, more so in oncology where many new drugs are used that are yet to have adequate safety information from long term studies.
The subjective terms we found were used in the abstract, conclusion, or discussion (or in the “Research in context” box in Lancet and Lancet Oncology). These are arguably the most widely read sections in a research paper and may make a lasting impression on readers, who often lack the time to read the results section for further information. Although we focus on randomised trials, the use of subjective terms to describe harms is also common in phase I or II non-randomised studies as well as in conference presentations. The use of such terms in non-randomised studies is particularly concerning because readers do not have a control to make comparisons. No data are available on whether the harms reporting in oncology trials is worse than in other specialties, but the under-reporting of harms in trial publications is a well known problem irrespective of discipline.67
We consider the lack of harms reporting and the use of subjective terms to describe harms to be poor reporting practice. The CONSORT statement for reporting of harms has a table listing common poor reporting practices.8 The first item reads: “Using generic or vague statements, such as “the drug was generally well tolerated” or “the comparator drug was relatively poorly tolerated.”
All trial reports should avoid using vague and subjective terms to describe the harms of interventions. The trade-offs between benefits and harms will vary, and though benefits might outweigh the risks, no cancer drugs are completely “safe,” so we propose that this term should not be used.
Our study supports other evidence that reporting of adverse events is poor in cancer drug trials, with some studies failing to report the incidences of severe, serious, and fatal adverse events. These events should be documented in all trial reports.
Although brevity may be cited as one of the reasons for using general terms to describe toxicities in conclusions or abstracts, we propose two more accurate ways to tackle this problem.
The first is to ask patients about acceptability. All trials of cancer drugs could collect data from patients on whether they consider the treatment toxicities are tolerable or acceptable. The abstract conclusions could then state “64% of patients in the trial considered the drug to have tolerable toxicities” rather than using non-objective statements. Non-randomised trials could also use this approach.
A second solution is to report quality of life. For cancer drugs, quality of life information is an indirect indicator of harms and is also an important measure of clinical benefit. Thus, instead of a statement such as “toxicities were manageable,” the report could conclude that there was “no effect on patients’ quality of life” or that “quality of life was improved,” based on objective assessment using validated tools.
However, quality of life reporting in cancer drug trials may also be subject to the risks of spin. For example, even though adjuvant sunitinib after resection of high risk renal cell cancer worsened quality of life in the S-TRAC trial, it was reported as “Patients on sunitinib did report increased symptoms and reduced [health related quality of life], but these changes were generally not clinically meaningful, apart from appetite loss and diarrhoea, and were expected in the context of known sunitinib effects.”9 In another example when olaparib did not improve the prespecified primary quality of life analysis in patients with ovarian cancer, this was reported as “not having a significant detrimental effect.”10 Furthermore, many randomised trials of cancer drugs do not report quality of life end points and negative quality of life information is reported less often than positive outcomes.11
Some trials already report harms more transparently. For example, a recently published trial of rituximab plus lenalidomide versus rituximab plus chemotherapy reported in its abstract conclusion that “the safety profile differed in the two groups.”5 Although this statement is not very informative, it is at least an objective description and readers can look at the adverse effect profiles and frequencies for themselves. Another trial abstract concluded: “The rate of high-grade adverse events in the cabozantinib group was approximately twice that observed in the placebo group.”12
Medical journals can also help to improve reporting of harms in cancer drug trials. The use of subjective terms must be discouraged, especially in the abstracts and conclusions. Editors and reviewers should ask for detailed harms data and encourage authors to report numbers and incidence rather than the vague statements to describe the harms. As readers, physicians and patients should look at the toxicities data in the tables rather than trust generalised terms. Proper risk-benefit assessment of any cancer drug should be made with actual harms and efficacy data, and not based on general concepts of safe, tolerable, or intolerable.
Many reports of cancer drug trials use subjective terms to describe harms, especially in abstracts and conclusion
Vague and subjective terms can lessen the perception of harm and influence decisions about treatment
All cancer trials should fully report adverse events and avoid subjective terms
Assessments of quality of life or asking patients about acceptability of a treatment would provide a better guide for treatment
Contributors and sources: BG conceptualized the study. BG, TS, and KH collected the data and all authors participated in data verification. All authors participated in discussion and interpretation. BG wrote the first draft of the manuscript, which was revised and approved by all the authors. BG has a keen interest and has widely published in cancer policy. He also has an interest in clinical trial reporting of cancer drugs and hosts a monthly blog critiquing on reporting practices of major cancer drug trials. The authors are also interested in safety reporting in cancer drug trials and have previously published together on serious and fatal adverse events of sorafenib.
Competing interests: We have read and understood BMJ policy on declaration of interests and declare that BG is an adviser to the BMJ analysis team. YA has received research funding and personal fees from drug companies but none in relation to this work.
Provenance and peer review: Not commissioned, externally peer reviewed.