Intended for healthcare professionals


Challenges of independent assessment of potential harms of HPV vaccines

BMJ 2018; 362 doi: (Published 24 September 2018) Cite this as: BMJ 2018;362:k3694
  1. Lars Jørgensen, researcher,1,
  2. Peter Doshi, assistant professor2,
  3. Peter Gøtzsche, professor1,
  4. Tom Jefferson, researcher1
  1. 1Nordic Cochrane Centre, Rigshospitalet, Copenhagen, Denmark
  2. 2University of Maryland School of Pharmacy, Baltimore, Maryland, USA
  3. Correspondence to: T Jefferson tj{at}

After three years of trying to access trial data for HPV vaccines, Lars Jørgensen and colleagues find current transparency policies unfit for their purpose

Key messages

  • Public confidence in interventions such as vaccines relies on comprehensive, independent, and accurate assessments

  • Clinical study reports contain more information than journal publications but are harder to access

  • Only half of potentially eligible reports for a systematic review of HPV vaccines had been obtained after three years, and these were incomplete and contained redactions

  • Regulators did not have the full data and the manufacturers place restrictions on the dissemination of data

  • The process for releasing clinical study reports should be improved to make it faster and more complete

Since the registration of the first human papillomavirus (HPV) vaccine (Gardasil) in 2006, HPV vaccination has been rolled out across the globe.1 There are currently three registered HPV vaccines: GlaxoSmithKline’s Cervarix and Merck’s Gardasil and Gardasil 9. The vaccines are mostly given to healthy adolescents for the prevention of HPV related diseases, such as cervical cancer, making them an important public health intervention.1

In the late 2000s, reports of potential harms associated with HPV vaccines, including postural orthostatic tachycardia syndrome (POTS) and complex regional pain syndrome (CRPS), began appearing in the media (see supplementary data on,23 followed by reports in scientific journals.456 Both syndromes are conglomerates of signs and symptoms and their diagnoses are complex.

In response to concerns about potential harms, the European Medicines Agency carried out an investigation in 2015 and concluded there was no evidence of a relation between HPV vaccination and the two syndromes.7 Independent research8910 and systematic reviews111213 drew the same conclusion. However, we believe there is reason to be cautious. EMA did not do the assessment itself but relied on the vaccine manufacturers’ analyses of the underlying harm data.14 Furthermore, the systematic reviews were based exclusively or predominantly on journal publications, which often are limited by publication and other reporting biases.15 We have previously shown reporting bias of the HPV vaccine studies; in our analysis one third of the completed HPV vaccine studies were not published and only about half of completed studies posted results on

Although regulators can request the underlying data (such as clinical study reports with case report forms and serious adverse event narratives) for all a manufacturer’s trials, regulators usually conduct in-depth analyses of only some of the trials and do not use systematic review methods. For EMA’s investigation into a potential link between HPV vaccines and POTS and CRPS, the manufacturers included only half of the studies likely to contribute data.16 EMA concluded that the “benefits of HPV vaccines continue to outweigh their risks,”14 even though many cases of POTS and CRPS were unrecognised or under-reported by the vaccine manufacturers.1718

Best way to assess potential harms

Using journal publications as the source of data for systematic reviews often has serious limitations, especially for harms. For example, a systematic review that compared journal publications with unpublished reports—across 10 drug interventions—found that between 43% and 100% (median 64%) of the harms were missing from the journal publications compared with the unpublished reports.15 One of the included studies of psychiatric drugs, which compared journal publications with clinical trial register entries, found that about half of the deaths and suicides were missing in the journal publications.19

Drug companies describe their trials extensively in clinical study reports, which are included in licensing applications to regulators. Clinical study reports are structured according to international guidelines20 and provide much more detail than journal publications. For example, the publication for one HPV vaccine trial (NCT00122681) is 14 pages long,21 whereas its corresponding clinical study report is more than 7000 pages.22 However, systematic reviews rarely use clinical study reports.

A systematic review of randomised trials can identify even relatively rare harms, provided the trials are large enough and have adequate follow-up. We thought that the HPV vaccine trials were sufficiently large and with long enough follow-up to potentially resolve whether POTS and CRPS could be causally linked to the HPV vaccines if all the data could be obtained.23 We therefore decided to conduct a systematic review using the clinical study reports and their patient level data with serious harms narratives and case report forms, which provide the most detailed information on harms.23 However, most of the data are not publicly available and must be requested.

Accessing clinical study reports

In May 2014, we requested the HPV vaccine clinical study reports from EMA. Since 2010 it has had a policy that, “access to documents or parts thereof may be granted whenever an over-riding public interest in disclosure can be identified by the Agency.”24 However, EMA initially denied our request on the grounds that it “would undermine the protection of commercial interests.” We appealed, arguing that the public interest argument for a global public health intervention like the HPV vaccines was overwhelming. Subsequently, EMA approved our request and began to release the clinical study reports in September 2014.

Through our searches as part of our ongoing systematic review,23 we had identified 206 comparative studies, 48 of which we judged to be industry studies likely to have clinical study reports potentially eligible for our systematic review.16 However, EMA informed us that it held clinical study reports for only 29 industry studies, which meant that, even in a best case scenario, it would be unable to provide us with clinical study reports for all the studies we wished to assess. After three years, we had obtained just 18 clinical study reports (62% of EMA’s 29 reports), of which 12 were eligible for our review.

EMA released over 35 000 pages for the 18 clinical study reports (table 1) in 61 batches. Unfortunately, the reports still lacked important sections, such as protocols and serious harms narratives, and most reports contained redactions of allocation numbers, vaccine batch numbers, and study centre and participant ID numbers (fig 1). Only three reports included completed case report forms.

Table 1

Details of EMA’s release of 18 clinical study reports from September 2014 to July 2017*

View this table:
Fig 1
Fig 1

Example of a redaction applied by EMA to study centre numbers of a clinical study report (Cervarix study HPV-014)

EMA’s release of the documents in so many batches made it difficult to keep track of the data, with clinical study reports often divided across several files and across batches. For example, one study report (HPV-008) of 4263 pages was released in 17 files across seven batches over 12 months.

In January 2017, we were approaching our data lock date of 1 July 2017 for our systematic review23 and sent EMA a prioritised list of 11 clinical study reports from the largest and longest HPV vaccine trials that would most benefit our systematic review. We also asked EMA to speed up the release of the reports. EMA subsequently informed us that the slow release was due to the complexity of our request, a high volume of data requests they were handling—in 2017, industry submitted 379 of 865 requests25—and too few staff, and that the clinical study reports were, “released [to us] as submitted by the company except for the redactions that might have been applied,” confirming that for some studies, the clinical study reports that industry provides to EMA are incomplete (eg, missing appendices).

As the release was slow we considered obtaining clinical study reports directly from the vaccine manufacturers. However, Merck requires that researchers do not disclose data to third parties,26 and GlaxoSmithKline grants access to complete trial data only through a portal that prohibits the download and public distribution of data. These policies conflicted with our aim to make the underlying data for our systematic review publicly available,23 and we decided not to pursue this route. Although GlaxoSmithKline publishes versions of its clinical study reports on its trial register, the reports often lack serious adverse event narratives and case report forms, and the data on serious adverse events in the reports we downloaded were heavily redacted (fig 2).

Fig 2
Fig 2

Example of a redaction applied by GlaxoSmithKline to serious harms in a clinical study report (Cervarix study HPV-015)

From our incomplete data we identified several areas that needed clarification, especially the choice of comparator and the clinical evidence regarding the effects of the adjuvant used in the vaccines. We put our questions to the manufacturers and regulators. GlaxoSmithKline answered most of our questions, but Merck answered just one of eight satisfactorily. For example, we asked why it did not use an inert placebo injection in any of the Gardasil 4 trials. An inert placebo most closely replicates the real life choice people must make on vaccination (that is, to have vaccination or not), and use of an adjuvant in the control group could have made it harder to detect any harms from the vaccine. Merck responded with a four page description of the properties of aluminium salts but provided no explanation for its choice of a non-placebo comparator. After 14 months, EMA has not responded to our inquiries.

Ultimately, we ended up including clinical study reports for 24 of the 48 potentially eligible studies using 12 reports obtained from EMA and 16 obtained from GlaxoSmithKline; for four trials we obtained reports both from EMA and GlaxoSmithKline. The 24 studies represented about 80% of the participants. The reports’ incompleteness and redactions meant that our systematic review will be limited by reporting bias, which we had hoped our systematic review could reduce.23

Better process

There are signs of hope for more transparency and sharing of trial data: the European Union court recently ruled that whole clinical study reports cannot be considered commercially confidential27; the Food and Drug Administration (FDA) launched a pilot programme to release clinical study reports28; and Health Canada has said it will begin sharing trial data.29 But much work remains to increase the validity of systematic reviews that use clinical study reports and other regulatory trial data to reduce reporting bias.

In our view, independent researchers ought to be able to obtain complete and unredacted clinical study reports within a reasonable time frame without too many constraints or limitations—especially when potential serious harms are reported after regulatory approval. In October 2016 we complained to the European ombudsman about the problems with getting reports for our research, but the ombudsman judged in early 2018 that EMA’s actions were “reasonable” and did not constitute maladministration (see supplementary data on

The slow release and high demand reported by EMA may warrant extra staff dedicated to releasing clinical study reports. However, because of EMA’s workload and staff loss during Brexit, it recently scaled back its data sharing policies, limiting one to EU citizens and temporarily suspending publishing new data packages under the other policy.30 Nevertheless, since public interests ought to trump commercial interests, we believe that independent researchers should be granted priority for access to clinical study reports over industry.

Redaction policies also need to be reconsidered so that benefits and harms can be fairly assessed. US survey data indicate that most people are willing to have their data shared with independent researchers.31 While it is important that participants remain anonymous and efforts to ensure an acceptably low risk of identification should be maintained, liability for re-identification could rest with those who assess the clinical study reports and be punishable by law.

We notified the ombudsman of some of our recommendations (such as replacing patient ID numbers with novel ID numbers), but the ombudsman supported EMA’s argument—that replacing patient ID numbers would not sufficiently reduce the risk of patient identification—and judged EMA’s allocation of staff (12.5 full-time equivalents handling requests) as “reasonable.”

We encourage regulators and industry to enhance their release of clinical study reports to independent researchers. In particular, they should respond to future requests by providing a detailed list of the clinical study reports they hold (including which parts of each report are available) and an estimation of when the reports will start to be released and how long it will take. Most importantly, regulators should release complete and coherent clinical study reports.

In her decision, the ombudsman concluded that EU rules on access to documents “are ill-suited to the purpose of making (large amounts of) scientific data available to researchers.”32 The rules clearly need amending. Urgent changes are essential for open and transparent assessment of the harms and benefits of interventions.


We thank EMA and GlaxoSmithKline for making clinical study reports available. We thank GlaxoSmithKline for providing answers to our questions.


  • Contributors and sources: PG and TJ are experienced systematic reviewers. TJ and PD were coauthors of the first Cochrane review based on clinical study reports. LJ is an evidence based medicine critic at the Nordic Cochrane Centre and is coauthor of several articles on HPV vaccine assessment. This article originated from three years of preparatory work we carried out as part of a systematic review of HPV vaccines. All authors contributed to the conception, drafting, and revision of the article and approved the final submission. TJ is the guarantor.

  • Competing interests: We have read and understood BMJ policy on declaration of interests and declare the following interests. PD and TJ are recipients of a grant from the Laura and John Arnold Foundation to run a RIAT Support Center and were corecipients of a UK National Institute for Health Research grant (HTA–10/80/01 Update and amalgamation of two Cochrane reviews: neuraminidase inhibitors for preventing and treating influenza in healthy adults and children). They are also in receipt of a Cochrane Methods Innovations Fund grant to develop guidance on the use of regulatory data in Cochrane reviews. PD sued the attorney general of Canada to get access to data on HPV vaccines. He has received funding from the American Association of Colleges of Pharmacy for a study to analyse written medical information on the possible harms of statins. PD is also an associate editor of The BMJ and an unpaid member of the IMEDS steering committee at the Reagan-Udall Foundation for the FDA, which focuses on drug safety research. TJ is occasionally interviewed by market research companies about phase I or II drugs. He has acted as an expert witness in litigation cases related to the antiviral oseltamivir and influenza vaccines and been a scientific adviser to a legal team acting on oseltamivir. TJ was a member of three advisory boards for Boerhinger Ingelheim and a member of an independent data monitoring committee for a Sanofi Pasteur clinical trial of an influenza vaccine. From 1994 to 2013, TJ was the coordinator of the Cochrane Vaccines Field. He is coholder of a Jean Monnet Network Grant. He is an unpaid collaborator to the project Beyond Transparency in Pharmaceutical Research and Regulation led by Dalhousie University and funded by the Canadian Institutes of Health Research. LJ and PCG have no competing interests to declare.

  • Provenance and peer review: Not commissioned; externally peer reviewed.