Covid-19 vaccine trial protocols released
BMJ 2020; 371 doi: https://doi.org/10.1136/bmj.m4058 (Published 21 October 2020) Cite this as: BMJ 2020;371:m4058Linked Feature
Will covid-19 vaccines save lives? Current trials aren’t designed to tell us
Read our latest coverage of the coronavirus outbreak
The ongoing phase III trials for covid-19 vaccines are some of the most consequential randomised trials ever done. In September, following months of campaigning for greater openness,12 four manufacturers made their full study protocols publicly available.3456 The publications create a rare opportunity for “real time transparency” in which the conduct of clinical trials is opened to public scrutiny while the studies are still under way.
Whatever the results ultimately show, public release of these protocols—each over 110 pages containing far greater detail than was previously available—enables a more scientific, deliberative, and inclusive trial process. While protocol transparency is not new—and is already common for high impact trials7—transparency in real time is an important development and a great win for public trust.
We may not like what we read, but with real time sharing of full protocols comes an unprecedented space for translating critique into action to improve trial design mid-stream. And there is much to critique.
The first question is whether the right endpoints are being studied. Contrary to prevailing assumptions (including those of a former Food and Drug Administration commissioner8), none of the vaccine trials are designed to detect a significant reduction in hospital admissions, admission to intensive care, or death.9 Rather than studying severe disease, these mega-trials all set a primary endpoint of symptomatic covid-19 of essentially any severity: a laboratory positive result plus mild symptoms such as cough and fever count as outcome events (table 1). These studies seem designed to answer the easiest question in the least amount of time, not the most clinically relevant questions.
Details from the trial protocols for covid-19 vaccines3-6
We shouldn’t be surprised. Regulators not only agree with these endpoints but have prespecified the “success” criterion as 50% efficacy against the primary endpoint (with a confidence interval that includes efficacy as low as 30%).10 Considering that these are relative risk reductions, absolute risk reduction will be important to assess once results are in, especially to assess benefit-risk profiles in healthy populations.
Built for speed
The second question is whether the trials are recruiting people at high risk. The study protocols suggest this intention, but sample size calculations were apparently based on the expectation of very low event rates in the control arm—around 1% a year for some trials. If these predictions are accurate, 99% of participants receiving a placebo will not develop symptomatic covid-19 over the next year, leading to large “number needed to treat to benefit” estimates.
Presumably, these were conservative planning estimates that justify larger trials capable of delivering efficacy results fast. The focus on speed fits with the prespecified final “event driven” analyses at roughly 150 total events, with many interim analyses specified even earlier.
But the low event rate assumptions foster an impression that vaccines are being tested on people at low risk of getting covid-19—and even lower risk of severe disease—who may be unrepresentative of populations prioritised to receive an approved vaccine. Manufacturers should be encouraged to provide real time data on the baseline characteristics of recruited participants.
Sixty years after influenza vaccination became routinely recommended for people aged 65 or older in the US, we still don’t know if vaccination lowers mortality. Randomised trials with this outcome have never been done.9 Observational studies with results in both directions can be cited, and without definitive randomised evidence the debate will go on. Unless we act now, we risk repeating this sorry state of affairs with covid-19 vaccines.
Eyes on deck
The covid-19 vaccine protocols should be scrutinised by the widest possible readership, to open a critical discussion of many questions about their design and conduct. These include why children, immunocompromised people, and pregnant women have been excluded from most trials; whether the right primary endpoint has been chosen; whether safety is being adequately evaluated; and whether gaps in our understanding of the clinical implications of pre-existing T cell responses to SARS-CoV-2 are being addressed.11
We still have time to advocate for changes to ensure these trials investigate the questions that most need answers. While the notion of “moving the goalposts” during ongoing trials may raise concerns, meaningful, scientifically valid, transparently reported amendments to protocols that are made to improve the value of trial results should always be welcomed.
Critical appraisal of clinical evidence traditionally occurs after the publication of studies, and generally finds much to criticise. Early release of full trial protocols offers a historic opportunity for the democratisation of science. The covid-19 vaccine trials may not have been designed with our input, but it is not too late to have our say and adjust their course. With stakes this high, we need all eyes on deck.
Footnotes
Competing interests: I have read and understood BMJ policy on declaration of interests and declare I have been campaigning for greater clinical trial data transparency of covid-19 products.
Provenance and peer review: Commissioned; not externally reviewed.
This article is made freely available for use in accordance with BMJ's website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may use, download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.
https://bmj.com/coronavirus/usage