Intended for healthcare professionals

Analysis

EMA must improve the quality of its clinical trial reports

BMJ 2011; 342 doi: https://doi.org/10.1136/bmj.d2291 (Published 25 May 2011) Cite this as: BMJ 2011;342:d2291
  1. Corrado Barbui, associate professor of psychiatry,
  2. Cinzia Baschirotto, researcher,
  3. Andrea Cipriani, lecturer in psychiatry
  1. 1WHO Collaborating Centre for Research and Training in Mental Health and Service Evaluation, University of Verona, Verona, Italy
  1. Correspondence to: C Barbui, Section of Psychiatry and Clinical Psychology, Policlinico G B Rossi, Piazzale L A Scuro 10, 37134 Verona, Italy corrado.barbui{at}univr.it
  • Accepted 15 March 2011

Corrado Barbui, Cinzia Baschirotto, and Andrea Cipriani find that the results of phase III studies are poorly and inconsistently documented in the EMA’s drug assessment reports. Better reporting would make them more useful for doctors, researchers, and consumers

In Europe, new drugs are approved or rejected on the basis of the results of studies carried out by the manufacturer and submitted to the European Medicines Agency. However, the transparency of the approval process has been criticised.1 2 Calls have been made for the European Public Assessment Report (EPAR), a summary of the grounds for granting marketing authorisation (box 1), to include additional information on critical points examined and discussed during assessment such as whether a drug is approved by majority vote, the reasons for the minority’s opposition, and decisions of other licensing bodies, in a timely and user friendly format.1 2

Box 1: Assessment of new drugs in Europe

European Medicines Agency (www.emea.europa.eu)

The European Medicines Agency (EMA) is an agency of the European Union responsible for the scientific evaluation of medicines developed by drug companies for use in EU countries. The EMA’s decisions on new or old medicines relating to changes in therapeutic indications, approval, suspension, or withdrawal of a product have to be accepted by all EU members.3

European public assessment report

Once the EMA has given marketing authorisation for a drug, it publishes a scientific assessment called the European Public Assessment Report (EPAR). The EPAR, written in agreement with the industry,4 summarises the documentation produced by the manufacturer and describes procedures that led to the EMA approval. EPARs are published on the EMA’s website after commercially confidential information has been deleted (www.ema.europa.eu/ema/index.jsp?curl=/pages/medicines/landing/epar_search.jsp&murl=menus/medicines/medicines.jsp).

Surprisingly, no criticism has been raised about the quality of the information that is currently available, especially the quality of reporting of phase III studies. This is an important aspect considering that new drugs are approved on the basis of the results of studies carried out by the manufacturer. Accessing the results of these studies may be relevant for doctors, who need to know the size of effect of newly licensed drugs for prescribing reasons, and for researchers involved in systematic reviews, who may need to use EPARs to access unpublished data for meta-analyses because not all the study reports that are submitted to regulatory agencies are published in the international literature.5 Consumers and the wider public may also want to check the basis for approval of new drugs.6 We use the example of drugs for psychiatric disorders to highlight deficiencies in current reporting.

EMA reporting standards

The quality of reporting of results of phase III studies in EPARs has been emerging as a challenging problem for researchers conducting systematic reviews of drugs for psychiatric disorders.7 8 We examined the EPARs of psychiatric drugs (see bmj.com for references) for information on four key aspects of trial reporting highlighted in the 2010 CONSORT statement9—that is, the number of patients randomised to each treatment arm, losses during follow-up (plus the reasons), number of patients included in the primary outcome analysis, and absolute numbers and effect size (with precision) for the primary outcome analysis (box 2).

Box 2: Search strategy and methods used to extract information from EPARs on psychiatric drugs

Search strategy

We did a systematic manual search of documents published on the EMA’s website (January 2010). We identified the European Public Assessment Reports (EPARs) of approved drugs for schizophrenia, acute mania and prevention of relapse, agitation in schizophrenia and bipolar disorder, major depressive disorder, generalised anxiety disorder and insomnia.

Data extraction and presentation

Working independently and in duplicate, two reviewers read the EPARs and identified the studies described as phase III clinical trials. For each of these, data were extracted on the following basic aspects of trial reporting: number of patients randomised per treatment arm, losses during follow-up, and number of patients included in the primary outcome analysis. Availability of the results of the primary outcome analysis was also investigated in terms of (a) absolute numbers for each treatment arm: number of subjects with the outcome of interest/total number of subjects (dichotomous outcomes); total number of subjects, mean end point or change score at the outcome of interest, standard deviation or standard error (continuous outcomes) and (b) effect size with its precision. We used a tabular approach to data presentation.

The EMA approved eight drugs from 2004 to 2009 for 15 psychiatric indications (table 1). Of the 70 phase III randomised trials described in the EPARs, 34 (49%) reported the number of patients allocated to each treatment arm, 19 (27%) reported drop-outs with reasons, 30 (43%) the number of patients analysed for primary outcome, and nine (13%) efficacy in terms of absolute numbers. Only six of these nine trials gave an effect size with its precision. This lack of data and erratic reporting made it impossible to use meta-analysis to calculate a summary measure of the overall treatment effect for any of the newly licensed drugs. Table 1 shows that the quality of reporting is improving for some items (numbers of patients in each treatment arm and included in the efficacy analysis) but not for others (drop-outs, efficacy results).

Table 1

 Availability of information required in the CONSORT 2010 statement from European Public Assessment Reports for phase III studies of drugs approved for psychiatric disorders

View this table:

Although this example cannot be used to make a general conclusion about the quality of reporting of all phase III studies on the EMA’s website, the results for drugs for psychiatric disorders are described in a way that is of little value. It is not possible to ascertain the degree of difference between active compounds and placebo, and consequently, it is difficult to make an informed judgment on the evidence that makes a new drug eligible for approval. Percentages are often reported without absolute numbers and without denominators, mean change scores are almost always reported without a measure of precision, and descriptive presentations of study findings may omit key figures, such as the number of patients randomised to each treatment arm. For example, in the EPAR for melatonin, one phase III clinical trial is described as follows: “523 patients entered the study, 453 were randomised and 334 were included in the Full Analysis Set, 169 in the Circadin group and 165 in the placebo group.”10 The number of patients randomised to each treatment arm cannot be determined from the description. This is not a minor point; without this information it is not possible to calculate any summary measure of treatment effect, and the potential exclusion of randomised patients from the analysis may result in biased estimates of treatment effects.11

Lack of consistency is another big problem. One EPAR may contain the required details for some studies but not for others, and no background logic seems to explain this inconsistently organised reporting.

Better reporting is possible

The provision of information on the websites of regulatory authorities in different countries has been shown to vary widely. A 2008 survey of six national regulatory agencies (United States, Canada, UK, France, Australia, and New Zealand) and the EMA showed that only the US Food and Drug Administration, the Canadian and French agencies, and the EMA provided public assessment reports for each new drug approved.12 The FDA information included comprehensive reports of clinical trials, whereas other agencies provided only abbreviated and summarised information. The FDA documents, however, have more recently been described as lengthy, inconsistently organised, and weakly summarised, making the information they contain practically inaccessible.13 In June 2009 the commissioner of the FDA announced a major transparency initiative with the goal of better explaining the FDA’s actions by providing information that supports clinical medicine, biomedical innovation, and public health.14 This initiative has already led to several draft proposals, although the FDA has not yet implemented new guidance on reporting trials in FDA reviews.

We argue that the EMA should develop and implement a similar transparency initiative. As initial step, a more informative description of the results of phase III studies would require no additional costs and would not require the release of any proprietary information. We suggest that the EPARs should include, for each phase III study, a tabular description of basic information on patient disposition and outcomes, together with the trial identification number that uniquely identifies a specific study (such as that from clinicaltrial.gov or similar). Table 2 shows an example template with a minimum set of information to properly describe the results of phase III studies of drugs for psychiatric disorders. The template could be adapted for other clinical areas and circumstances. Such data abstractions would be a balanced compromise between the ideal situation of having access to all original trial data and the current situation of having access to sparse and incomplete information.

Table 2

Template for better reporting of patient disposition and results of the primary outcome analysis in EPARs

View this table:

Some drug companies have already started using templates to disseminate trial results. GlaxoSmithKline (GSK), for example, was required in 2004 to provide access to all published and unpublished data from GSK sponsored clinical trials in an easy and user friendly format (www.gsk-clinicalstudyregister.com), and these data have already contributed to systematic reviews and meta-analyses,15 16 with an obvious added value for the scientific community.

Although a tabular description of basic information for each phase III study would mainly be designed to meet the needs of researchers doing systematic reviews and meta-analyses, the EMA might also attempt to target broader audiences of doctors and consumers. Doctors may benefit from summaries of randomised evidence presented in the EPARs, and these summaries could be developed following the example of the “summary of findings” table in Cochrane reviews. These tables provide key information concerning the quality of evidence, the magnitude of effect of the intervention examined, and the sum of available data on all important outcomes. A randomised trial showed that summary tables improve understanding and rapid retrieval of key findings when compared with reports with no table.17 Similarly, consumers may be interested in a concise and straightforward summary of the benefits and side effects of the new drug, and this summary could follow a structure similar to that of the drug facts box, a one page table quantifying outcomes with and without the new drug developed in the US. Providing consumers with a drug facts box has been shown to improve their knowledge of the benefits and side effects of prescription drugs.18

It should be highlighted, however, that these or similar reporting templates for doctors and consumers can be produced only if the results of phase III study are fully available. Thus our suggested tabular description of basic information on patient disposition and outcomes would represent a minimum but essential prerequisite for any further development of data presentation. This further development could be done by independent organisations or by the EMA.

Better provision of information on the EMA’s website would improve its value with relatively little effort. Doctors would have the opportunity to know the magnitude of effect of newly licensed drugs, authors of systematic reviews and meta-analyses would access trial results that might never be published in scientific journals, and consumers would have the chance to closely monitor the whole drug approval process aiming for continuous improvement.

Notes

Cite this as: BMJ 2011;342:d2291

Footnotes

  • Contributors and sources: C Barbui and A Cipriani have long experience in issues related to the methodology of systematic reviews and meta-analyses. C Baschirotto’s MD dissertation focused on European regulations for granting marketing authorisations. C Barbui and C Baschirotto read the EMA’s documentation and analysed the studies. C Barbui conceived the paper and wrote the first draft. All authors contributed to the writing, revised the text critically, and approved the final version. C Barbui is the guarantor.

  • Competing interests: All authors have completed the unified competing interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare no support from any organisation for the submitted work; no financial relationships with any organisation that might have an interest in the submitted work in the previous three years; and no other relationships or activities that could appear to have influenced the submitted work.

References

View Abstract