Intended for healthcare professionals

CCBYNC Open access

Rapid response to:


Drug treatments for covid-19: living systematic review and network meta-analysis

BMJ 2020; 370 doi: (Published 30 July 2020) Cite this as: BMJ 2020;370:m2980

Rapid Response:

Re: Drug treatments for covid-19: living systematic review and network meta-analysis

I think several aspects of the way this living systematic review/meta-analysis is reported are highly problematic. The highly positive conclusion and presentation of hydroxychloroquine brings these particular aspects into sharp focus.

The infographic, abstract, and readers' note are sure to be the most-read part of this review: and from them, you would get the impression that the analyses were up-to-date with emerging evidence, as of 21 July. In fact, the evidence in the analyses is a month older and that's a lot in Covid-19 trial time.

The only clue at the top level of this review this it is missing the very large trial on hydroxychloroquine (and one of the only very large trials overall) known to the public since June [1], is if you examine the infographic closely enough to pick up on what "Upcoming" data sources means. That should be clearer to non-experts, not just in the fine-print and on a hover note. As that infographic isn't in the PDF, there is no clue in the highly-skimmed part of that version that the conclusions on hydroxychloroquine are known to have been affected by the bias of including atypical early results. That such a large volume of data is missing from this version isn't even listed as a limitation of the study.

The way 21 July is often chosen throughout this document instead of the date that really reflects the age of the evidence is problematic beyond the infographic as well. As too, is the design choice for the summary table, giving such a bright green "most beneficial" colour-coding to findings with low/very low certainty.

The message being sent by that colour-coded table is deeply misleading, even apart from the fact that it is so out-of-date on hydroxychloroquine. A strength of network meta-analysis is identifying superior treatments, but it can't do that when there isn't enough data to be even mildly certain: in that circumstance, a report should not give the impression to non-methodological experts that it has done so. That's even more critical when it's known only a small fraction of the evidence on an intervention is included in the analysis.

This is a design/methodological problem. That's demonstrated, I believe, in the risk that the next update may not entirely solve this problem, given that the report of the RECOVERY trial [2] doesn't include the outcome given a bright green "most beneficial" rating. There is something inherently absurd in a system that could potentially highlight an intervention shown in a dominating large trial to be "associated with an increased length of hospital stay and increased risk of progressing to invasive mechanical ventilation or death" [2] with a bright green "most beneficial" signal for "time to symptom resolution" based on some small outlier trials.

In a situation like this, either the entire analysis should have been held off, or at least analyses on hydroxychloroquine clearly flagged as out-of-date and not included in the summary table, with no statement about its benefits highlighted in the abstract.

Given how few large and/or high quality trials there are for treatments of this disease, the benefit of a network meta-analysis based on data published in journals or preprints from June 2020 was likely to be of marginal benefit anyway, so fast-tracking wasn't likely to be of great benefit. As it's reported, I think this review is a misleading addition to the debate over treatment for Covid-19 at the end of July 2020.



Competing interests: No competing interests

01 August 2020
Hilda Bastian
Writer, meta-scientist