Intended for healthcare professionals

Rapid response to:

Research News

Study claiming Tamiflu saved lives was based on “flawed” analysis

BMJ 2014; 348 doi: https://doi.org/10.1136/bmj.g2228 (Published 19 March 2014) Cite this as: BMJ 2014;348:g2228

Rapid Response:

Re: Study claiming Tamiflu saved lives was based on “flawed” analysis

A point-by-point response to Dr Jones’s critique by the authors of the Muthuri et al. (2014) paper follows:

Dr Jones’s critique: “A crude analysis of the data shows an increased risk of mortality associated with neuraminidase inhibitor treatment,”suggesting that the finding of a reduced risk of death was incorrect.

Author’s response: It is indeed the case that, based on simple number counts, more people in the antiviral treated group died compared to those who were not treated. However, this can be explained if people in the treated group had a higher baseline risk of dying as compared to the people in the untreated group. In this kind of work we encounter the issue of non-equivalent comparison groups.

Ideally, one would conduct a randomised controlled trial (experimental study) where equivalent patients are randomly assigned to treatment or placebo. This way if we observe any differences in patient outcome, we can be more confident that these could be attributed to treatment status alone. In a pandemic situation, it would have been unethical to randomly deny antiviral treatment to patients and indeed we uncovered no RCT data during our extensive search for data pertaining to the pandemic period. Thus, we only had the option of studying actual treatment practice and resultant patient outcomes during the 2009-10 pandemic.

In order to overcome the issue of comparing non-equivalent patient groups, we used statistical methods to ‘adjust’ for any patient differences to allow us to disentangle treatment effects from outcomes arising due to fundamental differences among patients. This is why the adjusted results are paramount not the crude (unadjusted) results which Jones has chosen to highlight. Notwithstanding, we considered it important to present the crude results in the interest of transparent scientific reporting.

Of rather more concern is the fact that Jones has ignored the clustering of effects by study centre in his calculations of the crude estimates. We amalgamated data from 78 studies; it is standard statistical ‘good practice’ to accounting for such clustering, and in our analysis we considered this to be essential.

‘Clustering of effects’ means that there may be differences at the study level such as differences in the way healthcare is provided, accessibility to treatment, payment for healthcare or prescriptions etc. that could introduce further differences (heterogeneity) in our study. This is why it is standard statistical practice to account for these study level differences by performing ‘clustered’ analyses, also known as multilevel modelling. As indicated in the paper, the crude and adjusted results were modelled using generalised linear mixed models to take into account the correlation among participants within the same study (clustering). These models were used to preserve the validity of our conclusions, since ignoring such correlation can lead to misleading clinical and statistical inferences. Where there is variation in the baseline event rate between the studies and the true effect size is clinically important (as within our study), simulations have shown naïve analyses, such as those performed by Jones, can bias effect estimates towards the null (Abo-Zaid, 2013). Therefore, it is not surprising that our valid results do not reflect the misleading results of Jones who, put simply, has undertaken the wrong analysis.

Dr Jones’s critique: The complex analysis does not take into account time-dependent bias.

Authors’ response: We have acknowledged the point about time-dependent treatment effects in our paper. Time-dependent treatment effects can impact findings such that treatment can appear to be favourable as compared to no treatment because of a bias termed ‘immortal time bias’ which is observed because patients who die early do not get an opportunity to receive treatment. This is precisely why, for the subset of our population, where dates of illness onset and antiviral administration were available, we used a time-dependent Cox regression shared frailty model. Even after using this approach we found an approximately 50% reduction in mortality associated with neuraminidase inhibitor (NAI) antiviral treatment.

Standard techniques for modelling treatment variables as time-dependent covariates involve splitting follow-up time into ‘time before the treatment’ and ‘time after the treatment’ (Kleinbaum and Klein, 2012). This is the approach we have followed for our survival analysis comparing NAI treatment to no treatment, in order to overcome immortal time bias. In essence then, we only include follow-up time after NAI treatment has been initiated to avoid counting apparent ‘survival time’ before NAI treatment was prescribed. This is further explained in the accompanying diagram.

Dr Jones’s critique: “The analysis that is reported to include NAI [neuraminidase inhibitor] treatment as a time-dependent exposure is incorrect, because the result is impossible, and the survival curves indicate a standard Cox regression has been fitted.”

Authors’ response: Jones would need to provide a more detailed explanation of why he thinks that our results are ‘impossible’. His assertion is predicated on the fact that time-dependent bias can only ever work in the direction of favouring treatment. In fact it was a frequent clinical observation during the 2009-10 pandemic period that in many instances, a diagnosis of influenza and the instigation of antiviral treatment occurred when the patient was relatively late in the illness and by this stage deteriorating rapidly; this would conceal a treatment benefit.

With regards the survival curves in our published paper, Figure 2 relates to a different survival analysis from the one reported in the text referring to the findings for treated versus non-treated patients. Figure 2 represents survival curves for treated patients only. The purpose of these survival curves is to explore the effects of treatment delay on survival. Therefore, to answer this rather different question, we do need to take into account the time between illness onset and NAI treatment initiation and a time-dependent analysis involving splitting of follow-up time into ‘time before treatment’ and ‘time after treatment’ as outlined in our previous point, is not relevant. Rather, the effects associated with time to initiation of treatment are explored by stratifying treatment by categories denoting ‘time to treatment initiation from illness onset’.

This has been made very clear in the manuscript with regards the analysis relating to treated patients only (emphasis added for the purpose of this rebuttal):

“When only treated cases were considered, there was an approximately 25% increase in the hazard rate with each day’s delay in initiating treatment up to day 5 as compared to treatment initiated within 2 days of symptom onset [adj. HR, 1.23 (95% CI, 1.18- 1.28)].”

Dr Jones’s call for release of our data for verification by independent researchers

Authors’ response: Finally, Jones asks the authors to release their data so that a full independent analysis can be done. Ideally, in the spirit of transparency, we would have a publicly accessible pooled dataset so that other researchers can validate our analyses. In reality though, we are subject to data sharing agreements with our data contributors and may not be given permission by individual research groups within the PRIDE consortium or their local ethics boards to share this data even in a pooled format. Our data sharing agreements include explicit clauses on this matter as follows:

“(The PRIDE study investigators) recognise the Depositor's rights in the Data. Save as expressly provided for in this Agreement, no rights to or property in the Data shall pass to the University and the Depositor reserves all such rights…shall not transfer, distribute or release the Data to any third party without the prior written consent of the Depositor”

We will revisit this discussion with all our data contributors once we have completed all our planned analyses outlined in the PRIDE study protocol (PROSPERO CRD 42011001273). We have however endeavoured to be transparent in the reporting of our methods and can make available our Stata code used for the analyses and log files to independent researchers who would like to scrutinise our approach on application to Professor Jonathan Nguyen Van-Tam (jvt@nottingham.ac.uk). We will always be happy to provide clarifications regarding our published findings, to the extent that the data allow.

Professor Jonathan Van-Tam (senior author and strategic lead for this study), on behalf of the Department of Health, England has already provided UK data from the FLUCIN hospital pandemic influenza surveillance cohort to Professor Chris Del Mar, Coordinating Editor, Cochrane Acute Respiratory Infections Group on the 5th of July 2012.

On the basis of our clarifications above, we entirely reject Jones’s assertion that our analysis was ‘flawed’. Our study has limitations inherent to observational data which we do not deny, and have clearly declared in the manuscript but within these constraints we have used the best possible analytical approach available to us and taken on board any further advice on statistical analysis from experts outside the PRIDE study collaboration, including a co-convenor of the Cochrane individual patient data meta-analysis methods group. In the absence of better evidence, our study provides highly credible evidence that should be considered by policy makers as they contemplate any independent decisions they make about replenishment of antiviral stockpiles as part of ongoing pandemic preparedness strategies.

In summary, in response to Jones’s critique of our analytical methods:
• A crude analysis of our data that ignores clustering of effects by study centres is inappropriate.
• We have taken into account time-dependent biases in our Cox regression shared frailty models and this reiterates the findings obtained from the generalised linear mixed models that NAI treatment was significantly associated with a reduction in mortality during the 2009-10 pandemic.
• The survival curves referred to by Jones relate to treated patients only to explore the effects of treatment delay on survival.

References

Muthuri SG, Venkatesan S, Myles PR, et al. (2014) Effectiveness of neuraminidase inhibitors in reducing mortality in patients admitted to hospital with infl uenza A H1N1pdm09 virus infection: a meta-analysis of individual participant data. Lancet Respir Med; published online March 19. http://dx.doi.org/10.1016/S2213-2600(14)70041-4

Abo-Zaid G, Guo B, Deeks JJ, Debray TPA, Steyerberg EW, Moons KGM, Riley RD. Individual participant data meta-analyses should not ignore clustering. Journal of Clinical Epidemiology 2013; 66(8):865-873.e4

Kleinbaum, D.G. and M. Klein, Survival Analysis: A Self-Learning Text. Third Edition ed. 2012, New York: Springer.

Competing interests: Co-authors of the paper being critiqued in this article.

26 March 2014
Puja R Myles
Public Health Specialist/Epidemiologist
Jo Leonardi-Bee, PRIDE research consortium investigators
University of Nottingham
Clinical Sciences Building, City Hospital, Nottingham, NG5 1PB