The comparative risk of thrombosis with thrombocytopenia syndrome or thromboembolic events associated with different covid-19 vaccines: an international network cohort study from five European countries and the US

compares the risk of thromboembolic and thrombotic events with thrombocytopenia in people vaccinated with different types of COVID-19 vaccines – mRNA vaccines (Comirnaty and Spikevax) and adenovirus vaccines (Vaxzevria and Janssen COVID-19 vaccine). The authors gave enough background and were clear about their objectives. The findings are similar to other studies the authors cite (basically, COVID-19 adenovirus vaccines increase the risk of thrombosis with thrombocytopenia syndrome when compared to mRNA vaccines). This work is highly important for informing decisions regarding which COVID-19 vaccines should be prioritized. The


14-Jun-2022 BMJ-2022-071594
The comparative risk of thrombosis with thrombocytopenia syndrome or thromboembolic events associated with different covid-19 vaccines: an international network cohort study from five European countries and the US **Report from The BMJ's manuscript committee meeting** These comments are an attempt to summarise the discussions at the manuscript meeting. They are not an exact transcript.
Detailed comments from the meeting: First, please revise your paper to respond to all of the comments by the reviewers. Their reports are available at the end of this letter, below.
Please also respond to these additional comments by the committee: 1) Please provide clear information about the actual total number of individuals included in each of your comparisons, reporting by country. This could be done as a table or figure and could be part of your supplementary material. This could be the equivalent of your Figure 1 but for each comparison/cohort. Response: We have included a new table providing the requested information for all of the study cohorts as Supplementary Table 13. 2) Regarding your Figure 1, please expand as to how the comparator group 'gained' 111,994 people as part of your matching strategy. Please expand on the matching strategy used in the paper. Response: We used a maximum ratio of 1-to-4 matching. That means when the target group is larger, 4-to-1 matching is allowed. We have expanded the explanation of this in the revised manuscript.
3) Currently, the results reported are inconsistent, sometimes presenting on UK cohort, others for US, etc. Please give tables and figures for the estimates of each cohort that were included in your pooled estimates (Report the point estimates per database as well as the measure of heterogeneity for the pooled estimates). Your supplementary Table 7 makes clear that pooled estimates were only available for a handful of comparisons and from a subset of the cohorts. This needs to be reported better in your Results section.

Response:
Thank you and apologies that our previous version was not clear enough. We have now summarized all information in Supplementary table 7 (Summary of diagnostics at the database-target-comparator level).
As per the study protocol, database-specific estimates with insufficient power (MDRR>5) were not reported, but were later included in the meta-analysis if covariate balance was achieved after propensity score matching. In Supplementary Table 13, we added the MDRR for the matched cohorts at each database-target-comparator-outcome level, together with an indicator of whether the estimate was used in the meta-analysis. This has been clarified in the revised manuscript.
4) The use of Random Effects or Fixed Effect Models should not depend on heterogeneity observed. Please adjust and present results based on a selected model while defending your choice. Sensitivity analyses are acceptable to check for robustness. Related, please provide estimates of heterogeneity linked to your pooled estimates.

Response:
Sorry for the confusion. We pre-specified per protocol that random effects models was to be used to pool results from each of the databases. We did therefore not decide this based on the observed heterogeneity. In addition, we did not meta-analyse databasespecific findings where substantial heterogeneity was observed, using a pre-specified threshold I2>40%. Again, this was pre-specified in our study protocol, which can be found in the EMA-hosted EU PAS Register: https://www.encepp.eu/encepp/viewResource.htm?id=46886 5) Please be consistent in your reporting of effects. Your pooled estimates are only consistent with an effect of the AZ vaccine vs Pfizer on Thrombocytopenia. All other estimates are consistent with the null (e.g. the estimate presented for Janssen vaccination does not appear to be consistent with their interpretation). This might be due to lack of power which should be included in your discussion.

Response:
We have re-organized the results section following this logic: target-comparator combination, meta-analysis results by outcomes, and database-specific estimation. Discussion on the pooled results has been added as well. 6) A novel aspect of your work could be to estimate the rate of these major adverse events in a reference group (e.g. Pfizer). Assuming that this vaccine is not associated with an increased risk, it might be possible to refer to this as a baseline risk and estimate this for some of your age subgroups or at least by country (taking into account the age distribution of those present).

Response:
Thank you for the suggestion. We have now calculated empirically calibrated incidence rate differences and 28-day absolute risk differences for outcomes where a signal or marginal signal had been detected in our original IRR analyses. These are included in the revised manuscript text, and we believe this does indeed enhance the usefulness of our data.
7) The names of the vaccines are confusing and difficult to follow through the paper. Please consider changing them back to the generic names, or providing a box early in the paper with information on these.

Response:
Thank you for the suggestion. We changed all names to the generic name (ChAdOx1, BNT162b2, mRNA-1273, and covid-19 Vaccine Janssen BioNTech) in the revised manuscript.
8) It's disappointing that it only looked at a small piece of the picture. There is no comparison with thrombotic events from covid-19 itself, no breakdown into age groups and no analysis on the booster dose.

Response:
We did complete age-stratified analyses but did not include them in the previous version to simplify our message. These are added and covered in the text of the revised manuscript where all three diagnostics were met (MDRR<5, all SMD for covariate balance after matching <0.1, and calibration available). Sex-stratified results had also been completed, and have been added in the revised manuscript as Supplementary Tables 12, 13. We have highlighted in the text the only analysis where a relevant excess risk was identified, i.e. the observed increase in ATE risk in Janssen vaccinees (vs mRNA recipients) in ages 20-29 yo.
For comparison with Covid-19 infection, we believe that there are existing studies on this topic, including publications in the BMJ (e.g. https://doi.org/10.1136/bmj.n1931). As explained above, ours is a study commissioned by the European Medicines Agency, where this comparison was not requested or considered a relevant analysis. Finally, regarding boosters, these were not available in most of the contributing datasets by the time we extracted data for this study (December 2021). We agree that future studies on booster doses will contribute to public health evidence.
9) It seems to assume the individual receives the same vaccine for 2nd dose. What if they received a different 2nd vaccine? Response: In this study, we only included patients who received two doses of vaccines of the same brand. We have clarified this in the Methods section of the revised manuscript.
10) This is still of interest but may be mostly of historical interest by now. Could the primary message be made clearer? Response: We respectfully disagree on this point. First, this is to our knowledge the first analysis of comparative safety of adenovirus-vs mRNA-based COVID-19 vaccines. After a literature review, we only identified an analysis of comparative safety, but this focussed on Pfizer vs Moderna (both mRNA-based) vaccines, and conducted using only one cohort from US veterans. This manuscript was published less than one month ago (https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2793236 ). Although previously used methods including self-controlled case series and historical comparisons were useful, the use of an unvaccinated cohort or person-time has substantial limitations in terms of confounding and systematic error (Schuemie M et al. Front Pharmacol 2022;Li X et al. Front Pharmacol 20222). Conversely, the use of an active comparator (here, mRNA vaccines) has a number of methodological advantages, including better resolution of confounding by indication or severity (J Lund et al. Curr Epidemiol Rep 2015). In addition, active comparator analyses provide a more clinically relevant comparison in the current scenario, answering the question "which of the available vaccine/s is safer?" (in terms of thrombosis/coagulopathy), rather than "is vaccination safe, compared to no vaccination?".
Additionally, ours is also the first international study on this topic. The inclusion of multinational data from North America and multiple European countries provided a unique opportunity to test the robustness of our findings in different healthcare settings and populations. The use of a common data model and standardized analytics facilitated this, and has been recognised by multiple stakeholders including regulatory agencies as a way to increase reproducibility and transparency in real world evidence generation. The use of international data also allowed us to analyse both adenovirus-based (ChAdOx1 and Janssen), and to therefore confirm that the observed potential increased risk of TTS or TTS-VTE is a class rather than vaccine-specific effect.
Third, in an unprecedented example of transparency and aiming to maximise reproducibility, all our analytical code is available in a public repository, enabling independent researchers to peer review our code, and to replicate our analyses in their own data: : https://github.com/oxfordpharmacoepi/ROC22_CovVaxComparativeSafety/tree/main/CovVaxComparativeSafety Finally, our study uses novel analytics not previously implemented for the proposed research question, including methods to account for observed confounding (large-scale propensity score matching) as well as unobserved (negative control outcomes and empirical calibration). These methods are acknowledged and recommended in regulatory guidelines for drug safety research, including the most recent version of the ENCEPP "Guide on Methodological Standards in Pharmacoepidemiology": https://www.encepp.eu/standards_and_guidances/methodologicalGuide.shtml We have now been highlighted this in the revised version of the manuscript.
11) The calibrated IRR is sometimes confusing. For example, sometimes the IR is higher than the reference group but the calibrated IRR is <1 (e.g, the IR for Thrombocytopenia in CPRD >> the IR for Vaxzevria 2nd dose was higher than the Comirnaty 2nd dose, but the calibrated IR is <1). The original article on calibration (Ref# 31) talks about calibration of p-values, not of IRR.

Response:
We have added the requested reference on confidence interval calibration.
12) The background rates also varied in these databases, which the authors have reported before (BMJ 2021;373:n1435). therefore, it would require a more elaborate discussion as to whether (i) the background risk had any impact on the effect estimates, and (ii) if it was appropriate to pool the effect estimates.

Response:
We have reworded in the Discussion section and added more text on the heterogeneity of databases and the proposed use of meta-analysis. 13) How do the rates in this study differ from those reported elsewhere, say, in BMJ 2021;374:n1931 Response: In BMJ 2021;374:n1931, self-controlled case series analysis was conducted and the incidence rate ratios were presented. While background incidence rates using 2015-2019 were provided in the supplementary data, we failed to find post-vaccine event rates in the mentioned study. We compared effect estimates from this paper with ours in the revised version of our Discussion section.
14) The number of events were small for individual outcomes and a 30% higher risk may mean very different risk based on the baseline rates. So, it would be more appropriate to report absolute risk differences, particularly by number of vaccines administered.

Response:
Thank you for this suggestion. We have now included incidence rate differences and 28day absolute risk differences per 100,000 for the events where a signal or marginal signal was observed in the original pre-specified analyses.
15) It's not clear (especially for non-methodologist readers) what "Comparisons were limited to those with sufficient statistical power" means.

Response:
We have further explained this in the "Methods -statistical analysis" section.
16) The quality of databases is worrying. Table 1 shows the 7 databases have varied data quality and completeness, with only 2 having complete vaccination data and 1 having linked hospital treatment and outcome information (thus the capture of outcome events might not be complete).

Response:
We acknowledge that there are limitations in terms of exposure (vaccination) and outcome ascertainment, potentially of different magnitude in the different contributing databases. For instance, SIDIAP (ES) and CPRD (UK) provided complete vaccine coverage through linkage to national vaccination registry data, which is unusual and took time to accomplish. Reviewing previous literature it becomes obvious that some previous studies have not reported transparently on this important issue, but we believe it is important to acknowledge the limitations of the data utilised for these analyses. In addition, and in line with this, our choice of active comparator designs minimized the potential effect of incomplete vaccination information, as only vaccinated subjects were included and compared in our study.
Similarly, outcome misclassification is an issue in all "real world" (routinely collected) data sources, despite most authors not acknowledging this. This is much more of a problem when analyses are based on a single data source, minimising chances for acrossdatabase comparisons and replication analyses. While the capture of outcome events might not be complete, we do not expect outcome misclassification to be differential between vaccinated groups in our comparative safety analysis. This might be different and more problematic in previously published vaccine vs non-vaccine analyses, as unvaccinated subjects are potentially less likely to contact the healthcare system/s. All the included databases are recognised as valuable sources of information for epidemiological and regulatory research, and have all been previously used for similar research, including previous publications on vaccine safety in the BMJ (e.g. BMJ 2022;376: e068373, BMJ 2021;374: n1868, BMJ 2021373: n1435). We believe that each of the contributing data sources provides unique information from different healthcare settings, regions, and populations. Despite their specific limitations, access to these data allowed us to replicate our analyses internationally, adding reassurance to our findings.
We have expanded our text on database-specific limitations in the Discussion section. 17) Some important covariates are missing from PS matching, e.g. race/ethnicity, SES, BMI, smoking, etc, and since most covariates were extracted from medical records, their completeness and validity depends on the quality of the database.

Response:
Covariate presence differed by database. To maximise the completeness of our adjustments, we used large-scale propensity scores, which include all available confounders in each database. This strategy has been shown to account for both directly as well as indirectly measured confounders (https://arxiv.org/abs/2110.12235 ), and preferable to other propensity score estimation methods (https://academic.oup.com/ije/article/47/6/2005/5043131 ). Completeness of covariates and confounders is expected to be non-differential according to vaccine received.
In addition, we used robust diagnostics to minimise the inclusion of confounded results. One of the stop/go diagnostics in our study was the analysis of covariate balance after PS matching. All available covariates in the patient records/claims were assessed in these diagnostics, regardless of their inclusion in the propensity score equation. Any targetcomparator pair that failed to achieve the set covariate balance (SMD<0.1) for even a variable was excluded from our analyses as pre-specified in our published protocol.
In your response please provide, point by point, your replies to the comments made by the reviewers and the editors, explaining how you have dealt with them in the paper. Response: Thank you. We have provided a point-by-point response below. ** Comments from the external peer reviewers** Reviewer: 1 Recommendation: Comments: I read with interest this study. I commend the authors for their efforts. The study is very well conducted and in general I have no major comments and only 2 minor ones.
R1Q1. The first comment I have refers to the reported incidence rates for events (e.g. Line 282) that could be misleading to some readers if they remain reported as IR per 1000 person-years. I wonder if it would make more sense to report them in personweeks (although not very common) or in 28-day IR as the losses to follow up are likely to be low given the short time frame.

Response:
Thank you for the suggestion. While we understand that using person-weeks or 28-day incidence rate makes sense under the context, reporting incidence rate per 1000 personyear is more commonly used in previous studies, and therefore easier for comparing the results across different studies, and to previously published background rates.
R1Q2. Please emphasize that the risk refers to people receiving one vaccine versus another and not to the risk associated with the vaccines per se. In fact, to highlight the overall safety of the vaccines it may be helpful to report the incidence per doses administered. I strongly suggest you include this information. Response: Thank you for the suggestion. We have reported database-specific incidence rates of each event following each vaccine in Table 5. We have now also added incidence rate differences (per 1,000 person-year) and absolute risk differences (per 100,000 vaccination) between the different vaccines in Table 5. Further detail on rates is available in Supplementary Table 10.

Reviewer: 2
Comments: This paper is a detailed and comprehensive analysis examining risks of thrombosis associated with four different Covid-19 vaccines, combining results from large health databases from a number of European countries and the US. It is a major strength of the study that it was conducted across multiple countries and examined four different vaccine types (first and second doses only). Negative control outcomes -these are a useful addition and add to robustness of the current research.
Key points R2Q1. The authors should highlight more clearly what their research adds to the published literature as several papers reporting large observational studies have now been published on this topic, and it is unclear how their results compare with these other studies and whether their paper adds enough to what is already in the published literature.

Response:
We acknowledge that several studies have been conducted on this topic, yet most of them were based on a single database, and compared the risk of the events among people receiving one vaccine compared to a non-vaccinated population, and/or to COVID-19 patients. While most safety concerns regarding thrombosis events were raised among viral-vector vaccines, concerns about mRNA-based ones have been reported as well. This is therefore the first analysis of comparative safety of adenovirus-vs mRNAbased COVID-19 vaccines. We believe that this evidence does therefore contribute to the risk-benefit assessment of future vaccine and booster programmes by providing a more relevant answer at this stage of the pandemic, namely "what is the safest vaccine?" (in terms of thrombosis/coagulopathy) rather than the previously answered "are COVID-19 vaccines safe, compared to no vaccination?". The international analysis of all available vaccines in the contributing countries, and the use of robust novel methods should provide additional reassurance to our findings. We have clarified the novelty of our research in the Discussion section of our paper.
R2Q2. Covariates: Justification is needed for the selection of these covariates. Some are quite vague e.g. "medical history any time before cohort index" and further detail is needed. Response: In an attempt to minimize confounding by indication, we used large-scale propensity score methods, which included all available data in the patient records. The number of covariates differed by database and by vaccine/s studied, and included different specific variables depending on data availability. This method has been shown to reliably account for database-specific confounding in previous studies (J Lane et al. Lancet Rheumatol 2021) and in previous methodological research (see https://arxiv.org/abs/2110.12235 and https://academic.oup.com/ije/article/47/6/2005/5043131 as recent examples).
Full propensity score equations are available for review in a dedicated interactive web application, under the 'Propensity model' tab: https://dpa-pdeoxford.shinyapps.io/ROC22_CovVaxComparativeSafety_V2/ . We have clarified this in the Methods section of the revised manuscript, and created a new table (Supplementary  table 14) reporting on the top 10 variables in each propensity score.
R2Q3. Several published articles to date have shown differences in risks associated with vaccines between younger and older people (e.g. Andrews NJ, et al.  (2):e1003926). In the current paper it would be helpful to have additional analyses in age/sex sub-groups.

Response:
Thank you for the feedback. These analyses had been completed previously as per our published protocol (link), but not reported in detail in the previous iteration of the manuscript. We have now reported stratified results as suggested, and highlighted the most interesting findings. In line with some previous reports, we found an increased risk of ATE following some adenovirus vaccines when used in younger populations (age 20-29 yo).
R2Q4. The statistical analysis section needs to explain how missing data were addressed and how loss to follow-up was addressed in accordance with STROBE. Response: Thank you. We have explained both in the revised manuscript and accompanying STROBE checklist.
R2Q5. Figure 1 shows that a lot of people were dropped from the target population due to not being matched on propensity score, so the final numbers included in analyses are much smaller than those listed in the Abstract and the first paragraph of the Results. These final numbers for all vaccine comparisons should be included in the Abstract and in the main paper. The potential impact on results of dropping such large proportions of vaccinated individuals and on external validity/generalisability needs to be discussed.
We agree with the reviewer and have reported on these numbers in the revised abstract.
We have also added a new Supplementary Table 13 reporting cohort selection processes by database-target-comparator-outcome.
Finally, we have expanded the discussion on the use of propensity score matching, as well as impact on generalizability in our revised Discussion section.
R2Q6. The Results section is very detailed and quite complex to read and identify the key findings. It would be clearer if it presented the overall meta-analysis results and described any heterogeneity between countries as it is not easy to tell from the results whether there were consistent findings between countries.

Response:
Thank you for the suggestion. We have re-organized the result section using the following structure: targe-comparator pair, pooled estimates, and database-specific estimates. We hope this makes this section easier to read and understand for the BMJ readership.
R2Q7. The results are only reported in terms of relative rates but absolute and excess rates over 28 days post vaccination (e.g. per 1,000,000 vaccinated) would be informative and help with risk communication.

Response:
We fully agree with the reviewer that this is a better way to report our findings. We have therefore added estimates of incidence rate difference (per 1000 person-year) and 28day absolute risk differences (per 100,000 vaccination) in our revised manuscript.
R2Q8. Lines 347-348. The study from Scotland (ref 13) is rather cursorily dismissed in general terms which do not capture the main arguments in ref 37 or the related correspondence or specific features of the Scottish study, and it should be noted this study also included a self-controlled case series analysis.

Response:
We have revised the discussion on the study. R2Q9. Table 1 indicates incomplete data on vaccines and outcomes for some of the databases. This is a major limitation which needs further discussion in terms of likely impact on results. The limitation stated for detection of TTS (lines 384-388) is a major one.

Response:
We acknowledge that misclassification is a limitation of our analyses, albeit only for some of the databases. For example, SIDIAP (ES) and CPRD (UK) provided complete vaccine coverage through linkage to national vaccination registry data. Yet we believe this to be less problematic when using a concomitant (matched) comparative cohort analysis, where only vaccinated people are included. Similarly, outcome misclassification is less likely to be differential when two vaccinated cohorts are compared, instead of using unvaccinated or pre-vaccination time-at-risk as a comparator, as we know that healthcare systems recorded differently during different stages of the pandemic due to resource constrains. We have further discussed this in the Discussion section of our revised manuscript.
R2Q10. Paragraph in lines 389-395. It would enhance the paper and interpretation of results to give some information on the vaccination policies and approval dates in each country including which specific vaccines were recommended in which population groups. It should be noted that after propensity score matching to balance cohorts some of the higher risk groups might be excluded. It is also important given that as stated here the most vulnerable were vaccinated with the first vaccines approved that calendar time is accounted for and well balanced in the comparisons between vaccines, this needs to be stated more clearly in the results and emphasised here.

Response:
Thank you for the suggestion, we have expanded the discussion on the vaccination policy, and the impact on the propensity score matching, as well as generalizability of our results. We can reassure the reviewer that index month was one of the key covariates in our propensity score matching equations.
We clarified the use of minimal detectable relative risk in the revised manuscript. We pre-specified per protocol that any database-specific analysis with MDRR>5 would be underpowered and therefore not conducted.

R2Q17. Line 210 -<20% of what?
Response: We clarified this in the revised manuscript. Apologies that this was not clear in the previous iteration.
R2Q18. Line 213 -give some more detail for the Poisson regression. Did it account for the propensity score matching? Was an offset term included and how was it defined?

Response:
Since we only built the outcome models for target-comparator pairs with no observed unbalanced confounding after matching, we didn't include the propensity score in the outcome model. We didn't include an offset term in the Poisson model. R2Q19. Line 214. Empirical calibration is only described in very general terms. Further description is needed about how it works and explaining that it used the results for negative control outcomes.

Response:
In the revised manuscript, we expanded the description on using negative control outcomes to perform empirical calibration.
R2Q20. Line 330. The first sentence of the discussion should be removed. In a rapidly evolving area it is unclear whether the knowledge of the authors is based on a recent literature search. Also whilst this study was designed to give a direct comparison other published real-world studies allow indirect comparisons.

Response:
Thank you for the suggestion. We have modified the language in that sentence.
R2Q21. Numbers should be added to Tables 2-4. Response: We have fixed the tables in the revised manuscript.
healthcare system, and do therefore receive all information on their patients' health status, including hospital diagnoses. Finally, we expect the misclassification of outcomes to be non-differential between the vaccine groups, and should therefore have minimal impact on the estimation of relative risks.
We have acknowledged the potential measurement error in the outcomes as a limitation of this study. 5 -Regarding covariates, how was "medical history any time before cohort index" included in the propensity score? Can you please describe with further details and objectively the covariates used? Response: In our mapped databases, medical conditions were recorded in a table named "Condition," where all diagnoses from the original databases were included. In estimating the propensity score, we included the entire table in the LASSO regression. The model then selected the most relevant covariates associated with vaccine type, and included them in the propensity score equation. We have included the top 10 variables in each propensity score model in supplementary table 14.