Intended for healthcare professionals

Analysis

Evaluating covid-19 vaccine efficacy and safety in the post-authorisation phase

BMJ 2021; 375 doi: https://doi.org/10.1136/bmj-2021-067570 (Published 23 December 2021) Cite this as: BMJ 2021;375:e067570
  1. Christof Prugger, senior lecturer1,
  2. Angela Spelsberg, epidemiologist2,
  3. Ulrich Keil, professor3,
  4. Juan Erviti, researcher4,
  5. Peter Doshi, associate professor5
  1. 1Institute of Public Health, Charité-Universitätsmedizin Berlin, Berlin, Germany
  2. 2Comprehensive Cancer Centre Aachen, Aachen, Germany
  3. 3Institute of Epidemiology and Social Medicine, University of Münster, Münster, Germany
  4. 4Unit of Innovation and Organization, Navarre Health Service, Pamplona, Spain
  5. 5University of Maryland School of Pharmacy, Baltimore, Maryland, USA
  1. Correspondence to: C Prugger christof.prugger{at}charite.de

When covid-19 vaccines were first authorised, regulators required post-authorisation studies to tackle important uncertainties about efficacy and safety. But these studies may have little practical value unless there is greater engagement and scrutiny from the wider scientific community, argue Christof Prugger and colleagues

Expedited approval pathways have been increasingly used over the past 30 years to bring new medicines to market. The basic premise has been to give patients earlier access to medicines, often achieved by relying on less robust forms of evidence at the time of approval, such as showing efficacy against surrogate endpoints rather than patient outcomes.1

Expedited approvals are often coupled with requirements to conduct post-authorisation studies to confirm that the medicines safely provide the anticipated benefit. But a long history of concerns has emerged about the wisdom of shifting clinically important efficacy and safety assessments from before to after authorisation.1234 Post-authorisation studies often fail to deliver—lots of studies are never started, many take years longer than planned, and some fail to confirm pre-authorisation results. Evidence on relevant outcomes often remains inconclusive for several years,567 and post-authorisation safety events are seen more frequently for drugs with expedited approval.8 Regulators only rarely sanction companies for not adhering to post-authorisation study requirements, and drugs are only rarely withdrawn.2

Covid-19 vaccines are the most recent and prominent example of expedited regulatory approval. Here, we discuss the need to strengthen the design, conduct, reporting, and dissemination of post-authorisation studies, using covid-19 vaccines as a case study. We take a close look at improving the transparency landscape for post-authorisation studies required by the European Medicines Agency (EMA), focusing on 21 such studies for two mRNA vaccines, and argue that the appraisal of these studies should not be left to regulators alone. With active engagement of the wider scientific community, post-authorisation studies might finally deliver on their promise of providing answers to important questions in a timely fashion.

Limited evidence at time of conditional approval

The EMA granted conditional marketing authorisations for four covid-19 vaccines following the results of interim analyses of phase III randomised controlled trials. The European Union authorisations for the vaccines by Pfizer-BioNTech, Moderna, AstraZeneca, and Janssen are “conditional,” reflecting that, at the time of authorisation, less evidence than traditionally required for full approval was available on their safety and efficacy.9101112

At the two month mark, when the trials were assessed, manufacturers reported high efficacy relative to controls against laboratory confirmed covid-19 (of essentially any severity), but important unknowns remained. These included efficacy against SARS-CoV-2 infection, as well as severe covid-19 and the durability of efficacy after two years (the planned duration of the Pfizer-BioNTech pivotal trial (NCT04368728)).13141516

The US Food and Drug Administration listed important remaining unknowns in its review in December 2020: whether covid-19 vaccines reduce the risk of hospital admission, intensive care unit admission, severe covid-19, and mortality, as well as whether the vaccines are effective in populations at high risk of severe covid-19.1718 Groups of particular interest, such as older, chronically ill, or immunocompromised people, were under-represented in or excluded from trials.192021 Ongoing transmission in countries with high levels of vaccination highlights the importance of continued assessment of real world effectiveness.

Safety data on uncommon adverse events, as well as medium or long term harms of any frequency, were necessarily limited at the time of mass vaccine rollout, leaving some of the most important questions about efficacy and safety to the post-authorisation phase. Since authorisation and vaccine rollout, numerous studies have been published reporting high vaccine effectiveness at the population level and among particular groups such as healthcare workers and elderly people.222324252627. But many of these studies have important limitations including lack of data on hospital admissions, death, and high risk populations such as nursing home residents and people with comorbidities. Perhaps, more importantly, these studies were conducted outside of the regulatory framework—while they can be relevant they do not answer specific questions asked by regulators and might have limited influence on regulatory decisions.

Post-authorisation studies

After conditional marketing authorisation by the EMA, vaccine manufacturers Pfizer-BioNTech and Moderna agreed to carry out 13 and 8 post-authorisation studies, respectively,2829 to assess important unknowns including: risk of vaccine associated enhanced disease2829; effects in pregnant and breastfeeding women, people who were immunocompromised, frail, or with comorbidities or autoimmune or inflammatory disorders; potential interaction between different vaccines; and to provide long term safety data.

Conditional authorisation ensures that all post-authorisation obligations are legally binding and evaluated by the EMA. The requirements are codified in risk management plans written by the manufacturer and agreed by the regulator before authorisation. They are “an enforceable feature of the authorisation.”30 Risk management plans are publicly available documents detailing all planned and ongoing post-authorisation studies mandated by the EMA (see supplementary box on bmj.com). These post-authorisation (phase IV) studies contribute to the EU’s pharmacovigilance system alongside the more familiar spontaneous adverse event reporting schemes, EudraVigilance and the UK regulator’s Yellow Card Scheme.

Although the drug industry is officially responsible for conducting post-authorisation studies and meeting agreed deadlines for milestones (such as protocol development and study completion), the actual work of designing, conducting, and reporting these important studies can be done by various non-industry actors, such as academic institutions. The EMA also commissions academic and private sector partners to conduct some post-authorisation studies through the ACCESS (vaccine covid-19 monitoring readiness) project.31

Independent scrutiny

We think that researchers should be involved in both the planning and appraisal of post-authorisation studies. Independent scrutiny of regulator sanctioned studies can help close knowledge gaps on the efficacy and safety of medicines authorised through expedited pathways, by ensuring the right questions are asked and answered in a timely manner. Extra scrutiny would help improve the historically poor track record of required post-authorisation studies. Independent researchers can ensure transparency, evaluate study methods, monitor progress, and appraise results (box 1).

Box 1

Practical ways in which researchers can get involved in the appraisal of post-authorisation studies

  • Ensuring transparencyStudy documents, including the study protocol and interim or final clinical study reports, should be available in registration databases, such as the EU electronic register of post-authorisation studies,32 but they are not always accessible in practice.

  • Evaluating study methodsExamine study protocols; are they well designed? Are the right questions being asked? Are the methods for answering the research questions appropriate? Current guidelines for good study design are produced by the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance,33 and researchers can evaluate studies against these standards.

  • Monitor study progressAnalyse entries in study registers to consider whether important milestones, such as submission of interim analyses and final clinical study reports, are being achieved on time, as specified in risk management plans.

  • Examining resultsWas the study carried out and analysed as specified in the study protocol? For example, were pre-specified primary endpoints analysed? Are results transparently and consistently reported across different study reports (are findings in the study register fully consistent with those in a journal publication for example)?

RETURN TO TEXT

Patient and public participation in the process is also vital, particularly at the design stage. Only through patient and public involvement from the outset can we be sure that regulator mandated studies tackle the issues that matter most to patients. Moreover, specific informed consent should be obtained from participants in post-authorisation studies to allow sharing of individual patient data for independent scrutiny.

Independent scrutiny matters. Mayo-Wilson and colleagues, for example, found serious discrepancies in the reporting of trials of gabapentin for neuropathic pain and quetiapine for bipolar depression across different sources. These discrepancies among key trial characteristics, such as effect size and significance level, were large enough to influence the interpretation of trial results affecting drug approval and further research.34 Similar scrutiny of post-authorisation studies has also identified major inconsistencies and inaccuracies; for example, a post-authorisation study of dabigatran etexilate for patients with moderate renal impairment having hip or knee replacement surgery saw important changes in sample size and an interim analysis that occurred between the protocol and the final study report (which was late).3536

Contrary to the current process, whereby regulators and drug companies negotiate post-authorisation requirements behind closed doors, we argue for an open review of proposed study designs by independent scientists and patients, tackling issues such as study objectives, special populations of interest, study size and duration, primary and secondary outcomes, and the optimal time frame for reporting results. Such transparency is of even greater importance in view of emerging reports of “poor research conduct, lax data management, and a lack of regulatory oversight” at one of the contract research companies involved in a pivotal covid-19 vaccine trial.37

To illustrate the range of post-authorisation studies in need of third party scrutiny, we compiled a list of 21 studies specified in risk management plans after conditional authorisation by EMA of Pfizer-BioNTech and Moderna vaccines against covid-19 (see supplementary table on bmj.com). Study protocols or summary information were available for only five of the 13 Pfizer-BioNTech studies, and five of the eight Moderna studies. Two Pfizer-BioNTech studies aimed to inform the development of new versions of the vaccine or to study the adverse effects of a booster dose in healthy populations and immunocompromised patients (C4591001, BNT162-01 Cohort 13). Similarly, two Moderna studies aimed to test the effects of different doses or a booster dose on serious adverse events and immunogenicity as primary end points in healthy populations (20-0003, mRNA-1273-P201). One of the more interesting studies in the risk management plan (EU-PAS 40404) evaluates safety outcomes of four different vaccines in specific populations of interest (Pfizer-BioNTech, Moderna, AstraZeneca, and Janssen). EU-PAS 40404 is also the only study independent of vaccine manufacturers.

We could not locate either a protocol or summary information for the other eight Pfizer-BioNTech and three Moderna studies. Judging by the titles, some might provide information on hard outcomes such as severe covid-19 or hospital admissions (C4591011, C4591012, W1235284, W1235286). But the lack of publicly available study documents indicates that these studies have not been prioritised and remain at a very early stage.

The post-authorisation studies being prioritised by manufacturers seem to be those aimed at developing new vaccines or obtaining approval for additional doses of the current vaccines. The need for data on hard outcomes such as hospital and intensive care admissions or death in moderate or high risk populations is being overlooked.

Accessing study documents

Documents and data from post-authorisation studies can be accessed from databases or registers by searching for the trial identification numbers specified in each vaccine’s risk management plan (see supplementary table). Some of the most important studies for covid-19 vaccines are continuations of phase III trials, required by regulators. Fortunately, proactive release of key trial documents such as clinical study reports is expected in Europe following EU regulation 536/2014.38 Although not fully implemented yet, the EMA has started publishing documents from EU trials in the European Clinical Trials Register, making public important information on trial protocols, status, and clinical study reports.39 Likewise, clinical study reports and other documents that supported conditional market authorisation are now available on the EMA’s Clinical Data Publication website.40

The EMA plans to launch its new clinical trials information system in January 2022 as a single entry point for submitting clinical trial data in the EU. Clinical study reports and possibly other information in this system will be made public, subject to EU transparency rules. The EMA says that the system is already fully functional and that researchers should consider participating in its training programme on how to use it. The EMA is currently implementing a data analysis and real world interrogation network to generate timely evidence on the safety and effectiveness of medicines from healthcare databases.

To date, the EMA is the only regulator that provides access to data and documents from mandated post-authorisation studies, making summary data from protocols and study reports available through the EU electronic Register of Post-Authorisation Studies (EU PAS Register).32 The register allows for public access to administrative details, study objectives and main results, methodological details, and published documents including the full protocol with a signed checklist, conflicts of interest and the signed code of conduct, but not all these data are consistently provided. Basic study information from post-authorisation studies required by the EMA can also be found in other registration databases such as ClinicalTrials.gov.

When data and other information are not publicly available, freedom of information requests for any records held by the agency, including unpublished clinical data, can be made by EU residents using an online form on EMA’s website.41 Unfortunately, releases can be delayed,42 and documents are often released with redactions (to protect privacy or confidential business information) that may impede researchers’ efforts.

Although the US FDA and Health Canada also require post-authorisation studies, no proactive release of trial documents and data is yet in place. Health Canada does, however, provide easy, public access to clinical information related to studies underpinning authorisation.4344 To our knowledge, the UK Medicines and Healthcare Products Regulatory Agency does not have any plans for proactive release of data.

Making it happen

As post-authorisation studies mandated contribute considerably to assessment of the efficacy and safety of medicines and vaccines, particularly those authorised through expedited programmes, public access to data held by regulators is critical and should include patient level data, if available. Access would ideally be established at the planning stage to allow debate between regulators, marketing authorisation holders, and the scientific community throughout the process—from protocol preparation to submission of study reports.

Independent researcher engagement with regulatory studies largely remains an unfunded, voluntary effort. Funding bodies should consider giving a higher priority to these endeavours as money spent here would help improve the reliability, value, and timeliness of important post-authorisation studies. Journals also have a role in providing a place for third party critiques and analyses of post-authorisation studies, similar to the way many journals endorsed the restoring invisible and abandoned trials (RIAT) initiative, in which third party researchers reanalyse underlying study data independent of the original trialists.45

Both the scientific community and the public increasingly perceive the urgent need for independent evaluation of regulatory requirements.46 Rigorous evaluation of covid-19 vaccines’ safety and efficacy in the post-authorisation phase is critically important and increasingly possible thanks to strengthened transparency requirements for regulators. Without external scrutiny, we risk repeating the mistakes of the past—with many promises made but little follow through.

Regulatory agencies should continue to improve transparency by granting full access to all regulatory documents and available study data. And researchers should get involved in the independent evaluation of this material. Regulatory research (third party appraisal of industry funded studies mandated by regulatory authorities), independent of manufacturers and political interests, might relieve pressure on regulators47 and improve public trust by helping to ensure the safety, efficacy, and value of all medicines, including covid-19 vaccines—particularly those authorised through expedited regulatory pathways.

Key messages

  • Expedited approval of medicines by regulatory authorities often postpones the evaluation of important efficacy and safety endpoints until after medicines are widely available

  • For such medicines, well designed, conducted, analysed, and reported post-authorisation studies are vital to ensuring confidence that benefits actually outweigh risks

  • Covid-19 vaccines were widely administered following “conditional” authorisation based on short clinical trials, when important questions remained unanswered

  • Regulators require sponsors to carry out post-authorisation safety and efficacy studies, which often remain relatively unknown outside of specialist circles, and there is a history of insufficient compliance and regulatory oversight

  • Independent researchers must help scrutinise the design, conduct, data reporting, and overall transparency of post-authorisation studies mandated by regulators, particularly for global public health interventions such as covid-19 vaccines

Footnotes

  • Contributors and sources: The authors have extensively studied the role, practice, and contribution of post-marketing studies to pharmacovigilance in Europe. CP conceived the article, wrote the first draft and integrated input from co-authors and discussions in successive versions. All authors contributed to writing and editing. CP is the guarantor of the article.

  • Patient involvement: No patients were involved. Dissemination to this group is not applicable.

  • Competing interests: We have read and understood BMJ policy on declaration of interests and have the following interests to declare: PD has received travel funds from the European Respiratory Society (2012) and Uppsala Monitoring Center (2018); grants from the FDA (through University of Maryland M-CERSI; 2020), Laura and John Arnold Foundation (2017-22), American Association of Colleges of Pharmacy (2015), Patient-Centred Outcomes Research Institute (2014-16), Cochrane Methods Innovations Fund (2016-18), and UK National Institute for Health Research (2011-14); was an unpaid IMEDS steering committee member at the Reagan-Udall Foundation for the FDA (2016-2020), and is an editor at The BMJ. All other authors declare none.

This article is made freely available for personal use in accordance with BMJ's website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.

https://bmj.com/coronavirus/usage

References

View Abstract