Intended for healthcare professionals

Analysis

Call to improve transparency of trials of non-regulated interventions

BMJ 2015; 350 doi: https://doi.org/10.1136/bmj.h1323 (Published 27 March 2015) Cite this as: BMJ 2015;350:h1323
  1. Rafael Dal-Ré, director1,
  2. Michael B Bracken, professor and co-director2,
  3. John P A Ioannidis, professor and director3
  1. 1BUC (Biosciences UAM+CSIC) Program, International Campus of Excellence, Universidad Autónoma de Madrid, E-28049 Madrid, Spain
  2. 2Center for Perinatal, Pediatric, and Environmental Epidemiology, Schools of Public Health and Medicine, Yale University, New Haven, CT, USA
  3. 3Stanford Prevention Research Center, Department of Medicine, Department of Health Research and Policy, Stanford University School of Medicine, Department of Statistics, Stanford University School of Humanities and Sciences, Meta-Research Innovation Center at Stanford (METRICS), Stanford, CA, USA
  1. Correspondence to: R Dal-Ré rafael.dalre{at}fuam.uam.es
  • Accepted 17 February 2015

The public and clinicians require transparent, quality evidence for all interventions. Trials of non-regulated interventions are common, and efforts to improve their registration and publication compared with drug trials are overdue, say Rafael Dal-Ré, Michael Bracken, and John Ioannidis

Efforts to promote the availability of data from clinical trials have been led predominantly by regulators1 or drug companies2 and tend to focus on regulated interventions—that is, drugs, biologics, and medical devices. By contrast, trials of non-regulated interventions such as diets, exercise programmes, physiotherapy, surgical procedures, behavioural interventions, or complementary medicine have received much less attention.

To determine the benefits and harms of an intervention, randomised trial evidence is needed. These trials should be registered and published in an accurate and unbiased manner. Ideally clinical trial protocols and raw data should be made available to ensure transparency and for further use—for example, in individual patient data meta-analyses. These imperatives are well appreciated for trials of regulated interventions, but practices are lagging for non-regulated interventions.

All of us are likely to experience one or more non-regulated intervention in our lives. Yet many trials in these fields are small, underpowered, and lack quality safeguards such as appropriate randomisation, blinding, or choice of placebo or sham control.3 The results are often spurious. Very large, well conducted, trials on non-regulated interventions are rare, even for common lifestyle interventions such as diet and exercise programmes.4 5 Some trials of non-regulated interventions such as surgical procedures and behavioural interventions have intrinsic difficulties in their conduct,6 7 such as how to standardise the intervention. Non-standardisation leads to more heterogeneity and potential for bias.

Current mandates

In the United States the Food and Drug Administration is responsible for assessing all regulated interventions including drugs, biologics, and medical devices before they are licensed and commercially available. The European Medicines Agency has similar responsibilities in the European Union for drugs and biologics but not devices.

Before regulated interventions can be used in trials in the US sponsors must submit an application (an investigational new drug application or an investigational device exemption) to the FDA describing the protocols for the proposed clinical trials.8 9 In the EU trial protocols must be approved by the national regulatory agencies of the countries where the trial will take place.10 Both US and EU agencies review all clinical trial data submitted by the sponsor after the clinical development plan of the product is concluded.

Both US and EU regulations mandate registration of all phase II, III, and IV clinical trials of regulated interventions—before the trial starts in the EU and within 21 days of first participant enrolment in the US—and publication or public disclosure of trial results in a public registry with free access.10 11 12 Although initially developed to comprise trials on drugs and biologics, many registries—such as ClinicalTrials.gov and other registries belonging to the World Health Organization’s platform13—currently include trials of all types of intervention. Registration is the first step towards full trial transparency. It allows patients and clinicians to find what trials are being run for a given disease or condition. Among other benefits, registration helps to deter publication bias and outcome reporting bias.

Regulatory agencies have no role in trials with non-regulated interventions, and there are no US or EU regulations requiring the registration of these types of trial. In the absence of a mandated approach some organisations have set standards for the registration of trials of non-regulated interventions. For example, the International Committee of Medical Journal Editors,14 WHO,13 and the World Medical Association’s Declaration of Helsinki,15 as well as reporting guidelines, such as CONSORT16 and SPIRIT,17 and the AllTrials campaign,18 ask for prospective registration of trials of both regulated and non-regulated interventions. But these remain voluntary.

Quantifying trials of non-regulated interventions

The lack of transparency of trials of non-regulated interventions is evident in the pattern of trial registration on ClinicalTrials.gov, arguably the best source to investigate the registration of non-regulated intervention trials. We searched ClinicalTrials.gov on 16 August 2014 and found that only 12.4% (17 308 of 139 739) of registered intervention trials pertained to non-regulated interventions (table). This is a much smaller proportion than the 39% reported in published trials.19 These data are consistent with other evidence suggesting lower registration rates for trials of non-regulated interventions than for trials of regulated interventions—61% of trials published in clinical journals were found to be registered,20 compared with only 21% and 29% of trials published in psychological or behavioural21 and physical therapy22 journals, respectively.

Table

Quantifying trials assessing non-regulated interventions

View this table:

To estimate the number of trials of non-regulated interventions that are published we assessed a random sample of articles indexed in PubMed. A search for articles published in 2013 and tagged as “randomised controlled trial” yielded 31 772 items. We screened a 1% sample (318 items) by reading the title, abstract, and full text if needed and found that 210 of these items were indeed full text articles on randomised controlled trials. We found that 38% (80 of 210) of these trials assessed non-regulated interventions (table). Although this estimate is from a rough survey, it is close to the proportion estimated by Hopewell and colleagues (39%) for trials of surgical procedures, counselling, or lifestyle interventions among randomised trials published in 2006.19 So it seems that randomised controlled trials of non-regulated interventions make a major contribution to the literature, comprising roughly 40% of published randomised controlled trials.

Small sample sizes and no involvement of regulatory agencies are known risk factors for non-publication of trials of regulated interventions.23 Trials of non-regulated interventions are typically small—for example, in our random sample of 80 randomised controlled trials, the median number of participants was 92, and 22 trials enrolled fewer than 40 participants—and, by definition, lack the involvement of regulatory agencies. Thus, it may be that among launched trials far more than 39%, and perhaps the majority, pertain to non-regulated interventions.

External incentives needed

Experience with drugs and biologics24 tells us that we cannot rely solely on the willingness of investigators to publish all trial results with non-regulated interventions. External incentives may be needed to improve the registration and publication, and therefore transparency, of non-regulated interventions. Current trial registries may cater to the registration of trials of non-regulated interventions or may need to be amended to best capture some of their peculiarities. Some stakeholders in non-regulated interventions research may already be incentivising prospective registration and publication of trials, but others may need to move in this direction.

Editors can help by requiring registration of all trials to be published in their journals. Currently 28% of medical journals request trial registration,25 but most influential journals that mandate registration for publication are dominated by trials on regulated interventions. For example, of 151 trials published in New England Journal of Medicine, Lancet, or JAMA from January to June 2014, only 43 (29%) were on non-regulated interventions. Conversely, 11 of 16 trials (69%) published in The BMJ in the same period were on non-regulated interventions.26 Lack of enforcement of the registration requirement by journals leaves scope for non-registration and non-publication. Publishers could also have a role—Elsevier, for example, is assessing how to embed trial registration numbers into the peer review process.27 Both journals and publishers could require inclusion of the registration number and date registered as a prerequisite to successful online manuscript submission. However, this will not necessarily guarantee registration of a trial before it is started.

Since September 2013 research ethics committees in the UK have required registration as a condition of granting approval for trial protocols.28 Other countries could follow this example. Research ethics committees could also require that the registration number be provided in the information sheet. This would facilitate access to important trial information for participants and physicians.

Funders of trials could mandate prospective registration of non-regulated intervention trials and public access to their results. Initial funding might be suspended until trial registration, and a portion of final funding could be withheld until a manuscript has been submitted for publication. This is current practice for research financed by the UK National Institute for Health Research.29 The National Institutes of Health in the US has recently proposed a policy for registration and publication of results for all trials that it funds (wholly or in part), including those that are not subject to FDA regulation.30 31 Funding penalties or incentives may also be used for encouraging public access to raw data.

Professional associations could encourage policies on prospective registration of all clinical trials conducted by their members and public disclosure of results and respective data. All members might express in writing their commitment to follow these policies, possibly when becoming a member or for yearly membership renewals. Professional associations may also have a role in educating their members as to the importance of prospective registration and data access from all types of trial.

Coordinated effort

Regulated intervention trials are usually conducted by teams of trialists and companies that run many trials concurrently, whereas non-regulated intervention trials are often carried out by single teams, usually from academia. We need to test such interventions with a more coordinated effort, perhaps using multicentre or multinational consortia, as occurs with many regulated interventions. In contrast to regulated interventions, where industry typically controls the research agenda, such efforts should be managed by stakeholders without conflicts of interest. For example, in 2015 Stanford University will launch the Wellness Living Laboratory, a large cohort platform where informed participants (“citizen scientists”) are invited to choose whether they want to join any among dozens of randomisations pertaining to non-regulated interventions of lifestyle and other aspects of their wellness.32

The public, biomedical scientists, and clinicians require high quality evidence irrespective of the regulatory status of assessed interventions. Given the importance of non-regulated interventions, efforts to improve their transparency are overdue.

Key messages

  • Non-regulated interventions include surgical, behavioural, diet, and exercise interventions, among others

  • Most efforts towards trial transparency are led by regulatory agencies for regulated interventions, such as drugs, biologics, and devices

  • Some 38% of published trials are of non-regulated interventions compared with only 12% of registered trials, suggesting that the large majority of trials of non-regulated interventions are not registered

  • Journal editors, research ethics committees, scientific associations, and funders could do more to ensure prospective registration and publication of non-regulated trial results

Notes

Cite this as: BMJ 2015;350:h1323

Footnotes

  • Contributors and sources: RDR conceived the idea and wrote the first draft of the manuscript. All authors contributed substantially to the design of the article, as well as the analysis and interpretation of existing evidence. All authors drafted and revised the manuscript, approved the final version, and are accountable for it. All authors are engaged in clinical research and epidemiology and are supporters of activities aimed to ensure research transparency. RDR is guarantor.

  • Competing interests: We have read and understood BMJ policy on declaration of interests and have no relevant interests to declare. All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare no support from any organisation for the submitted work. The opinions expressed in this article are those of the authors and may not reflect the opinions of the organisations that they work for.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

References

View Abstract