Poor description of non-pharmacological interventions: analysis of consecutive sample of randomised trials
BMJ 2013; 347 doi: https://doi.org/10.1136/bmj.f3755 (Published 10 September 2013) Cite this as: BMJ 2013;347:f3755- Tammy C Hoffmann, associate professor of clinical epidemiology1,
- Chrissy Erueti, assistant professor1,
- Paul P Glasziou, professor of evidence-based medicine1
- 1Centre for Research in Evidence-Based Practice, Faculty of Health Sciences and Medicine, Bond University, Qld, Australia, 4229
- Correspondence to: T Hoffmann thoffmann{at}bond.edu.au
- Accepted 29 May 2013
Abstract
Objectives To evaluate the completeness of descriptions of non-pharmacological interventions in randomised trials, identify which elements are most frequently missing, and assess whether authors can provide missing details.
Design Analysis of consecutive sample of randomised trials of non-pharmacological interventions.
Data sources and study selection All reports of randomised trials of non-pharmacological interventions published in 2009 in six leading general medical journals; 133 trial reports, with 137 interventions, met the inclusion criteria.
Data collection Using an eight item checklist, two raters assessed the primary full trial report, plus any reference materials, appendices, or websites. Questions about missing details were emailed to corresponding authors, and relevant items were then reassessed.
Results Of 137 interventions, only 53 (39%) were adequately described; this was increased to 81 (59%) by using 63 responses from 88 contacted authors. The most frequently missing item was the “intervention materials” (47% complete), but it also improved the most after author response (92% complete). Whereas some authors (27/70) provided materials or further information, other authors (21/70) could not; their reasons included copyright or intellectual property concerns, not having the materials or intervention details, or being unaware of their importance. Although 46 (34%) trial interventions had further information or materials readily available on a website, many were not mentioned in the report, were not freely accessible, or the URL was no longer functioning.
Conclusions Missing essential information about interventions is a frequent, yet remediable, contributor to the worldwide waste in research funding. If trial reports do not have a sufficient description of interventions, other researchers cannot build on the findings, and clinicians and patients cannot reliably implement useful interventions. Improvement will require action by funders, researchers, and publishers, aided by long term repositories of materials linked to publications.
Introduction
Secret remedies—branded drugs whose ingredients were kept secret—were once common, until successful campaigns in the United States and United Kingdom in the early 20th century required labels to include all ingredients.1 This policy allowed independent evaluation of treatments and provided clinicians and consumers with the means to understand what they were considering using.
However, a treatment consists of more than the list of ingredients: the dose, frequency, monitoring, titration, mode of delivery, and duration of use can all influence efficacy and safety. Such details are often not well described in trials. For example, a recent analysis found that only 11% of 262 trials of cancer chemotherapy provided complete details of the trial treatments.2 The most common missing elements were dose adjustment and pre-medications, but 16% omitted even the route of administration of the drug. The completeness of descriptions of treatments may be worse for non-pharmacological interventions: one analysis found that 67% of drug treatment descriptions were adequate compared with only 29% of non-drug treatments. However, this analysis also included systematic reviews, which were less informative on average than were reports of individual trials.3
Unlike the bogus “secret remedies” of the 19th century, the current incomplete descriptions of interventions are likely to arise from poor communication and lack of awareness of the matter among authors and lack of attention by reviewers and editors. However, the consequences are similar: other researchers cannot replicate and build on research findings, and clinicians and patients cannot reliably adopt interventions shown to be useful.
To better understand what elements are most frequently missing from descriptions, and whether this is remediable, we aimed to assess the completeness of descriptions of non-pharmacological interventions in reports of trials published in major general medical journals, using a checklist to evaluate the completeness, and to assess whether missing details can be obtained by contacting authors of trial reports.
Methods
Search strategy and selection of reports of trials
We included all reports of randomised controlled trials of non-pharmacological interventions published in 2009 in one of the six leading general medical journals (based on ISI Web of Knowledge impact factor for 2010): New England Journal of Medicine, JAMA, Lancet, Annals of Internal Medicine, PLOS Medicine, and BMJ. We defined non-pharmacological interventions as interventions including surgery, technical procedures, devices, rehabilitation, psychotherapy, behavioural interventions, and complementary and alternative medicine.4
We excluded reports if they evaluated a pharmacological intervention, a screening programme, or a diagnostic technique (for example, endoscopy) or if they reported secondary analyses, such as an economic analysis of a non-pharmacological intervention trial for which the main results had been published elsewhere or before 2009. Reports were eligible if a non-pharmacological intervention had been compared with a pharmacological intervention or when a complex intervention included a pharmacological component if the major focus was a non-pharmacological component. When reports contained evaluations of more than one non-pharmacological intervention (such as by using three groups or a factorial design), we rated the descriptions of each intervention separately.
An experienced medical librarian searched PubMed in April 2011, using the restrictions of year (2009), publication type (“randomized controlled trial”), and journal title (the six chosen journals). Two authors (TH and PG) screened the 358 titles and abstracts retrieved, identified reports that might meet the inclusion criteria, and retrieved the complete and unabridged reports (n=138). Disagreements about unclear eligibility were resolved by discussion.
Rating of intervention descriptions
Before rating the reports in our study sample, we rated a sample of reports that met our eligibility criteria but had been published in 2008, discussed responses, and continued rating reports until high agreement was reached. Two authors (TH and PG) independently used a checklist to rate the description of the interventions in each eligible report. The checklist is based on the work of Davidson and colleagues and the CONSORT extension statement for non-pharmacological interventions and has been used, in a slightly modified form, in a previous study.4 5 6 The checklist contains eight items (table 1⇓). Seven of the items are rated as “Yes” (indicating that the element of the intervention had been clearly described) or “No” (not reported or not clearly described). Each item in the checklist had an explanatory statement that guided the raters in the scope and interpretation of the item. For “No” responses, we recorded detailed comments about what was missing. The eighth item was an overall question: “Is the description of the intervention complete?” Reports could score a “Yes” on this item only if all other items had been rated as “Yes.” We also routinely and systematically assessed whether the intervention details might have been described further in other sources (such as websites, online appendices, or reference materials). If so, we used these additional sources when completing the checklist items. Disagreements in ratings were resolved by discussion, with reference to the explanatory statement for the relevant item/s, between the two raters.
For all reports with any checklist items rated as having an incomplete description, the raters generated a list of questions about element/s of the intervention that were missing or unclear. We emailed these questions to the corresponding author. If no response was received within four weeks, we sent up to two reminder emails at four weekly intervals. When responses were received from authors, the raters collaboratively re-rated the relevant items as either “Yes—clear after author reply” or “No—not clear after author reply.”
Data analysis
We entered initial and follow-up ratings of reports and questions emailed to authors into a customised database. We used Excel to analyse data descriptively.
Results
Of the 138 reports for which we were able to obtain full texts, we excluded five (three economic evaluations and two long term follow-up results) and four reports contained evaluations of two non-pharmacological interventions (see supplementary figure). Our final sample thus contained published reports of 137 interventions from 133 trials. Table 2⇓ shows the journals in which the reports had been published and the categories of intervention that had been evaluated.7
Completeness of intervention descriptions
Figure 1⇓ shows, for each of the checklist items, the percentage of interventions that were clearly described in primary reports and after reply from authors of trial. Overall, 53 (39%) of the interventions were adequately described in primary reports, and this increased to 81 (59%) after contact with authors. The checklist item about intervention materials scored most poorly in primary reports (complete in 64 (47%) of interventions) but was also the item that showed the most improvement (complete in 126 (92%)) after reply from trial authors.
We sent questions to authors of reports about 88 of the interventions (from 84 trials), with a mean of 2.7 (SD 1.5, range 1-6) questions per author. We received no response from 25 authors after two reminders. On the basis of the responses from the 63 authors who replied, the overall rating changed from “No” to “Yes” for 29 of the interventions and remained as “No—description not complete” for 34 interventions (for example, requested materials were not sent or the response did not clearly answer the question). Table 3⇓ provides verbatim examples of incomplete reporting of descriptions of interventions from the primary reports.
Process of obtaining complete description of intervention—an example
Figure 2⇓ illustrates the key steps in the study methods and the process of obtaining a complete description of an intervention for one of the included reports. After the initial rating of this report,8 four items—setting, provider, procedure, and materials—were assessed as incompletely described, so the overall rating item (“Is the overall description complete?”) was “No.” We emailed the corresponding author with five questions (shown in fig 2⇓). On the basis of the responses, each of these items and the overall rating were changed to “Yes.” By contacting the author, we also learnt that even though the trial used a DVD that was not available, an iPhone app (in Dutch only) containing the exercises and exercise schedule was now available, as was a YouTube clip of the exercises. However, neither of these resources was mentioned in the 2009 report.
Materials used in interventions
Most of the questions that were sent to authors were about materials used in the intervention. We emailed 70 authors with at least one question about materials and received responses from 27 authors that enabled the rating of the materials item to be changed to “Yes.” Typically this was because authors sent copies of the materials used (for example, written materials used with study participants, such as patient education materials or staff training materials) or provided information about how to access intervention materials or further details (such as a website or journal article that was not referenced in the primary report). Of the other 21 authors who responded, the rating for the materials item, and the overall rating, remained as “No.” The box shows typical reasons for this.
Reasons given for study intervention materials being unavailable
Category of reason (number of authors providing a response in this category) and illustrative quotes from authors:
Materials not publicly available (9)
-
“Due to legal copyright restrictions at my university I am unable to send”
-
“Not publicly available because we based them on materials provided by our local government”
-
“Not publicly available—only to our trainers”
-
“Not yet—they will be made publicly available within two years”
-
“No it is not. Attached is a table of contents”
-
“The training materials from the trial are not online—we had no real reason to do that”
Corresponding author did not have copy of materials to send or could not provide further details about intervention (8)
-
“People originally in the position have moved on”
-
“I am unable to find . . . my old computer files”
-
“I’m afraid I no longer have access to those materials”
-
“I do not have it”
-
“I am not able to answer most of your questions. I was not involved with running the trial, only analysing and reporting on the QOL results after the data was collected”
-
“I can’t provide these”
Other (3)
-
“You will have to read the literature”
-
“No, is in Dutch”
-
“The [materials] are tailored, thus it is difficult to disseminate. We could send an example”
Materials were previously publicly available but no longer are (2)
-
“URL doesn’t exist anymore”
-
“We had been making it previously available, but need to update it, so are no longer”
Further intervention information contained on websites
For one third (n=46) of the interventions, a relevant website existed that contained further information about the intervention or at least some of the materials used in the intervention. Only 35 (76%) of these websites were mentioned in the report and, of these, just over half (19; 54%) were freely accessible. Many websites were behind a paywall, and for others the website address provided in the report no longer existed. Several interventions had relevant websites, but we learnt of these only by contacting the author or through internet search engines. Figure 3⇓ summarises these websites according to how we found out about the website’s existence (for example, mentioned in the original report, email from author), whether any website links provided worked, and whether the site content was accessible free of charge.
Discussion
More than half (61%) of the interventions assessed in this study were not described in sufficient detail in the published primary report to enable replication of the intervention in practice. This problem is partly remediable: a third of the incompletely described interventions could be completed by contacting study authors for further information. Obtaining this additional information took some effort: compilation of omissions, up to three emails, and subsequent piecing together of information from disparate sources. Clinicians wishing to use an intervention in practice are unlikely to invest this amount of time in obtaining the necessary details and materials. Even for trials in which the intervention was not effective, complete descriptions of interventions are important for other groups of research users. For example, this includes other researchers (who may wish to build on the trial or modify the intervention in some way) and systematic reviewers (who need details of intervention to assist with assessing and understanding heterogeneity).
Most interventions included some materials, such as written materials for educating patients or materials for training staff, without which the interventions cannot be used in practice. Despite their importance, just over half (53%) of the reports neither sufficiently described these materials nor gave details about how to obtain copies. Consequently, questions about materials were our most common question to authors. Comments from some authors suggested a lack of awareness of the importance of making intervention materials available. Other authors were reluctant to make materials available publicly (often owing to real or perceived concerns about copyright or intellectual property). Sometimes, corresponding authors no longer had copies of the materials or were uncertain about intervention details.
Several previous studies have documented deficiencies in the descriptions of interventions: an analysis of patient education interventions found that only 17% were replicable, with information about the content of sessions most commonly missing9; none of 11 trials of music therapy provided all necessary details10; and an analysis of 158 surgical studies found that only 41% provided some details of the surgical procedure.11 However, only one previous study seems to have included writing to authors to obtain additional information; a checklist was not used, and only a small sample of non-pharmacological interventions was included.3
Recommendations
The omission of essential information about interventions is a substantial, yet remediable, contributor to the enormous worldwide waste in research funding that occurs because research is unpublished or unusable.12 Reducing this waste will require action by funders, researchers, and publishers at multiple stages, including pre-submission, editorial review, and publication. However, missing details in the reports of intervention studies is part of a wider problem of non-replicable interventions in trials and systematic reviews. This has been the topic of a consensus meeting,13 which made several recommendations. Two of these are relevant to the problems identified in this research.
A key recommendation was that: “The reporting standards for interventions in trials (CONSORT, etc) and systematic reviews (PRISMA) should be improved and standardised (specific checklists).” The CONSORT extension statement for the reporting of randomised trials of non-pharmacologic treatment contains four intervention related items: precise details of the intervention, description of the intervention components (and where applicable, description of tailoring procedures), details of how interventions were standardised, and details of how adherence to the protocol by care providers was assessed.4 However, this CONSORT extension does not mention intervention materials, which we found to be the most poorly reported, and remediable, element. The mere existence of an appropriately detailed checklist is unlikely to be sufficient to improve quality of reporting. Few journals request the use of extension statements, and, even when endorsed by journals, subsequent adherence to them by published reports does not necessarily occur.14 Journal editors and reviewers have a responsibility to be aware of the importance of complete reporting of non-pharmacological interventions and to implement policies and processes to ensure that this occurs.
A second recommendation was that: “A stable ‘intervention bank’ should be established (e.g. videos, manuals, and fidelity tools linked to trial registration number) to overcome the problem of word restrictions in journals, etc.” Although hosting intervention materials on websites may seem to be a logical way of making them available, problems such as maintaining operational sites were already apparent in this sample of quite recent reports. One striking finding was that a third of the trials had a website with additional intervention information/materials, but a quarter of these were not mentioned and many were not freely accessible or the URL no longer worked. This suggests that responsibility for maintaining websites that contain intervention details should not lie solely with journals, authors, or their institutions, but instead with organisations that have greater stability and longevity. Similar to drug treatments in pharmacopoeias, an equivalent compilation for non-drug interventions is needed.15 16
Strengths and limitations of study
The strengths of this study include the wide range of non-pharmacological interventions, duplicate rating, and the verification of missing details and materials with authors. However, 28% of authors did not respond, limiting the completeness of this verification. A further limitation is that we selected reports only from leading general medical journals, and findings may not be generalisable. If the quality of reporting of interventions is similar to the quality of reporting of methodological features in randomised trials, which is better in journals with a higher impact factor,17 then our study may have underestimated the size of the problem. Our study may have also found an even larger problem of poor reporting if we had assessed whether the reports had evaluated or described the fidelity of the intervention, which has been identified as an additional element that should be included in descriptions of interventions.5
Further work needed
Further work is needed in several areas. Firstly, we need to better understand why authors do not provide more complete descriptions of interventions. Secondly, we need to develop the guidelines and tools to assist authors and editors in providing complete descriptions of interventions. Several of the extensions to the CONSORT statement have been motivated by a need to obtain better details of interventions, but these differing extensions need harmonisation to clarify what is generic and what is specific to different types of interventions. As many trials now publish a protocol, providing additional intervention detail in the published protocol may be one way of overcoming word limit restrictions in the trial’s primary paper. The recently published reporting guideline for trial protocols, the SPIRIT 2013 Statement,18 contains four items (11a-d) about the reporting of the intervention. Further work is needed to ensure that the primary paper makes explicit mention of all related documents (such as protocols, online supplementary material, and websites) so that readers can easily obtain a complete description of the intervention.
Although reporting standards are necessary, they are unlikely to be sufficient. Development and enhancement of tools to assist with describing interventions would be useful. One such tool is PatPlot,19 which provides a graphic depiction of the elements and sequencing of complex interventions. However, given the documented substantial deficiencies in reporting of interventions and the demonstration that it is partly remediable, authors and editors should take action now to reduce this waste in research.
What is already known on this topic
-
Incomplete descriptions of interventions render the intervention uninterpretable and unusable by clinicians, patients, and researchers
-
The completeness of descriptions of interventions may be worse for non-pharmacological interventions than for pharmacological interventions
What this study adds
-
Missing information is a common problem (occurring in more than 50%) in reports of non-pharmacological interventions
-
Details of intervention materials and procedures were the most commonly missing elements
-
The problem can be partially remediated by contacting authors of trial reports for missing details
Notes
Cite this as: BMJ 2013;347:f3755
Footnotes
-
We thank Iain Chalmers (editor, James Lind Library, Oxford, UK) and Susan Michie (professor of health psychology, University College London, UK) for reviewing the manuscript and providing helpful comments. We also thank the authors of included reports who responded to our questions.
-
Contributors: TCH and PPG were primarily responsible for study conception and design and for data analysis and interpretation. All authors had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. CE assisted with acquisition of the data and administrative support. TCH led the writing of the first draft of the manuscript, and all authors contributed to drafting and revising the manuscript. TCH and PPG are the guarantors.
-
Funding: TCH is supported by a National Health and Medical Research Council of Australia (NHMRC)/Primary Health Care Research Evaluation and Development Career Development Fellowship (number: 1033038), with funding provided by the Australian Department of Health and Ageing. PPG is supported by a NHMRC Australia Fellowship (number: 527500). The funders had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.
-
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: TCH is supported by a NHMRC/Primary Health Care Research Evaluation and Development Career Development Fellowship, and PPG is supported by a NHMRC Australia Fellowship; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
-
Ethical approval: Not required.
-
Data sharing: Data on the included trials and their ratings are available on request from the corresponding author at Tammy_Hoffmann{at}bond.edu.au.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.