Evaluating covid-19 vaccine efficacy and safety in the post-authorisation phase

Limited evidence at time of conditional approval

The EMA granted conditional marketing authorisations for four covid-19 vaccines following the results of interim analyses of phase III randomised controlled trials. The European Union authorisations for the vaccines by Pfizer-BioNTech, Moderna, AstraZeneca, and Janssen are “conditional,” reflecting that, at the time of authorisation, less evidence than traditionally required for full approval was available on their safety and efficacy.9101112

At the two month mark, when the trials were assessed, manufacturers reported high efficacy relative to controls against laboratory confirmed covid-19 (of essentially any severity), but important unknowns remained. These included efficacy against SARS-CoV-2 infection, as well as severe covid-19 and the durability of efficacy after two years (the planned duration of the Pfizer-BioNTech pivotal trial (NCT04368728)).13141516

The US Food and Drug Administration listed important remaining unknowns in its review in December 2020: whether covid-19 vaccines reduce the risk of hospital admission, intensive care unit admission, severe covid-19, and mortality, as well as whether the vaccines are effective in populations at high risk of severe covid-19.1718 Groups of particular interest, such as older, chronically ill, or immunocompromised people, were under-represented in or excluded from trials.192021 Ongoing transmission in countries with high levels of vaccination highlights the importance of continued assessment of real world effectiveness.

Safety data on uncommon adverse events, as well as medium or long term harms of any frequency, were necessarily limited at the time of mass vaccine rollout, leaving some of the most important questions about efficacy and safety to the post-authorisation phase. Since authorisation and vaccine rollout, numerous studies have been published reporting high vaccine effectiveness at the population level and among particular groups such as healthcare workers and elderly people.222324252627. But many of these studies have important limitations including lack of data on hospital admissions, death, and high risk populations such as nursing home residents and people with comorbidities. Perhaps, more importantly, these studies were conducted outside of the regulatory framework—while they can be relevant they do not answer specific questions asked by regulators and might have limited influence on regulatory decisions.

Post-authorisation studies

After conditional marketing authorisation by the EMA, vaccine manufacturers Pfizer-BioNTech and Moderna agreed to carry out 13 and 8 post-authorisation studies, respectively,2829 to assess important unknowns including: risk of vaccine associated enhanced disease2829; effects in pregnant and breastfeeding women, people who were immunocompromised, frail, or with comorbidities or autoimmune or inflammatory disorders; potential interaction between different vaccines; and to provide long term safety data.

Conditional authorisation ensures that all post-authorisation obligations are legally binding and evaluated by the EMA. The requirements are codified in risk management plans written by the manufacturer and agreed by the regulator before authorisation. They are “an enforceable feature of the authorisation.”30 Risk management plans are publicly available documents detailing all planned and ongoing post-authorisation studies mandated by the EMA (see supplementary box on bmj.com). These post-authorisation (phase IV) studies contribute to the EU’s pharmacovigilance system alongside the more familiar spontaneous adverse event reporting schemes, EudraVigilance and the UK regulator’s Yellow Card Scheme.

Although the drug industry is officially responsible for conducting post-authorisation studies and meeting agreed deadlines for milestones (such as protocol development and study completion), the actual work of designing, conducting, and reporting these important studies can be done by various non-industry actors, such as academic institutions. The EMA also commissions academic and private sector partners to conduct some post-authorisation studies through the ACCESS (vaccine covid-19 monitoring readiness) project.31

Independent scrutiny

We think that researchers should be involved in both the planning and appraisal of post-authorisation studies. Independent scrutiny of regulator sanctioned studies can help close knowledge gaps on the efficacy and safety of medicines authorised through expedited pathways, by ensuring the right questions are asked and answered in a timely manner. Extra scrutiny would help improve the historically poor track record of required post-authorisation studies. Independent researchers can ensure transparency, evaluate study methods, monitor progress, and appraise results (box 1).

Patient and public participation in the process is also vital, particularly at the design stage. Only through patient and public involvement from the outset can we be sure that regulator mandated studies tackle the issues that matter most to patients. Moreover, specific informed consent should be obtained from participants in post-authorisation studies to allow sharing of individual patient data for independent scrutiny.

Independent scrutiny matters. Mayo-Wilson and colleagues, for example, found serious discrepancies in the reporting of trials of gabapentin for neuropathic pain and quetiapine for bipolar depression across different sources. These discrepancies among key trial characteristics, such as effect size and significance level, were large enough to influence the interpretation of trial results affecting drug approval and further research.34 Similar scrutiny of post-authorisation studies has also identified major inconsistencies and inaccuracies; for example, a post-authorisation study of dabigatran etexilate for patients with moderate renal impairment having hip or knee replacement surgery saw important changes in sample size and an interim analysis that occurred between the protocol and the final study report (which was late).3536

Contrary to the current process, whereby regulators and drug companies negotiate post-authorisation requirements behind closed doors, we argue for an open review of proposed study designs by independent scientists and patients, tackling issues such as study objectives, special populations of interest, study size and duration, primary and secondary outcomes, and the optimal time frame for reporting results. Such transparency is of even greater importance in view of emerging reports of “poor research conduct, lax data management, and a lack of regulatory oversight” at one of the contract research companies involved in a pivotal covid-19 vaccine trial.37

To illustrate the range of post-authorisation studies in need of third party scrutiny, we compiled a list of 21 studies specified in risk management plans after conditional authorisation by EMA of Pfizer-BioNTech and Moderna vaccines against covid-19 (see supplementary table on bmj.com). Study protocols or summary information were available for only five of the 13 Pfizer-BioNTech studies, and five of the eight Moderna studies. Two Pfizer-BioNTech studies aimed to inform the development of new versions of the vaccine or to study the adverse effects of a booster dose in healthy populations and immunocompromised patients (C4591001, BNT162-01 Cohort 13). Similarly, two Moderna studies aimed to test the effects of different doses or a booster dose on serious adverse events and immunogenicity as primary end points in healthy populations (20-0003, mRNA-1273-P201). One of the more interesting studies in the risk management plan (EU-PAS 40404) evaluates safety outcomes of four different vaccines in specific populations of interest (Pfizer-BioNTech, Moderna, AstraZeneca, and Janssen). EU-PAS 40404 is also the only study independent of vaccine manufacturers.

We could not locate either a protocol or summary information for the other eight Pfizer-BioNTech and three Moderna studies. Judging by the titles, some might provide information on hard outcomes such as severe covid-19 or hospital admissions (C4591011, C4591012, W1235284, W1235286). But the lack of publicly available study documents indicates that these studies have not been prioritised and remain at a very early stage.

The post-authorisation studies being prioritised by manufacturers seem to be those aimed at developing new vaccines or obtaining approval for additional doses of the current vaccines. The need for data on hard outcomes such as hospital and intensive care admissions or death in moderate or high risk populations is being overlooked.

Accessing study documents

Documents and data from post-authorisation studies can be accessed from databases or registers by searching for the trial identification numbers specified in each vaccine’s risk management plan (see supplementary table). Some of the most important studies for covid-19 vaccines are continuations of phase III trials, required by regulators. Fortunately, proactive release of key trial documents such as clinical study reports is expected in Europe following EU regulation 536/2014.38 Although not fully implemented yet, the EMA has started publishing documents from EU trials in the European Clinical Trials Register, making public important information on trial protocols, status, and clinical study reports.39 Likewise, clinical study reports and other documents that supported conditional market authorisation are now available on the EMA’s Clinical Data Publication website.40

The EMA plans to launch its new clinical trials information system in January 2022 as a single entry point for submitting clinical trial data in the EU. Clinical study reports and possibly other information in this system will be made public, subject to EU transparency rules. The EMA says that the system is already fully functional and that researchers should consider participating in its training programme on how to use it. The EMA is currently implementing a data analysis and real world interrogation network to generate timely evidence on the safety and effectiveness of medicines from healthcare databases.

To date, the EMA is the only regulator that provides access to data and documents from mandated post-authorisation studies, making summary data from protocols and study reports available through the EU electronic Register of Post-Authorisation Studies (EU PAS Register).32 The register allows for public access to administrative details, study objectives and main results, methodological details, and published documents including the full protocol with a signed checklist, conflicts of interest and the signed code of conduct, but not all these data are consistently provided. Basic study information from post-authorisation studies required by the EMA can also be found in other registration databases such as ClinicalTrials.gov.

When data and other information are not publicly available, freedom of information requests for any records held by the agency, including unpublished clinical data, can be made by EU residents using an online form on EMA’s website.41 Unfortunately, releases can be delayed,42 and documents are often released with redactions (to protect privacy or confidential business information) that may impede researchers’ efforts.

Although the US FDA and Health Canada also require post-authorisation studies, no proactive release of trial documents and data is yet in place. Health Canada does, however, provide easy, public access to clinical information related to studies underpinning authorisation.4344 To our knowledge, the UK Medicines and Healthcare Products Regulatory Agency does not have any plans for proactive release of data.

Making it happen

As post-authorisation studies mandated contribute considerably to assessment of the efficacy and safety of medicines and vaccines, particularly those authorised through expedited programmes, public access to data held by regulators is critical and should include patient level data, if available. Access would ideally be established at the planning stage to allow debate between regulators, marketing authorisation holders, and the scientific community throughout the process—from protocol preparation to submission of study reports.

Independent researcher engagement with regulatory studies largely remains an unfunded, voluntary effort. Funding bodies should consider giving a higher priority to these endeavours as money spent here would help improve the reliability, value, and timeliness of important post-authorisation studies. Journals also have a role in providing a place for third party critiques and analyses of post-authorisation studies, similar to the way many journals endorsed the restoring invisible and abandoned trials (RIAT) initiative, in which third party researchers reanalyse underlying study data independent of the original trialists.45

Both the scientific community and the public increasingly perceive the urgent need for independent evaluation of regulatory requirements.46 Rigorous evaluation of covid-19 vaccines’ safety and efficacy in the post-authorisation phase is critically important and increasingly possible thanks to strengthened transparency requirements for regulators. Without external scrutiny, we risk repeating the mistakes of the past—with many promises made but little follow through.

Regulatory agencies should continue to improve transparency by granting full access to all regulatory documents and available study data. And researchers should get involved in the independent evaluation of this material. Regulatory research (third party appraisal of industry funded studies mandated by regulatory authorities), independent of manufacturers and political interests, might relieve pressure on regulators47 and improve public trust by helping to ensure the safety, efficacy, and value of all medicines, including covid-19 vaccines—particularly those authorised through expedited regulatory pathways.