Covid-19: Validity of key studies in doubt after leading journals issue expressions of concern

Doubts have emerged about the validity of widely cited research papers that have informed global health policy during the covid-19 pandemic.

On 2 June the Lancet issued an expression of concern1 relating to a study2 it published on 22 May suggesting that hospital patients with covid-19 being treated with hydroxychloroquine and chloroquine were at increased risk of dying and of ventricular arrhythmia than patients who did not receive the drugs. The study prompted the World Health Organization to halt part of its Solidarity trial of the drugs, although after reviewing the details its Data Safety and Monitoring Committee has now said there were no reasons to change the protocol.

Also on 2 June the New England Journal of Medicine also issued an expression of concern3 about a separate study4 by the same group of authors published on 1 May. This paper found that angiotensin converting enzyme (ACE) inhibitors and angiotensin receptor blockers were not associated with a higher risk of harm in patients with covid-19.

The journals’ action came as the Guardian newspaper published an investigation5 claiming that the two studies were both based on flawed data from a little known US healthcare analytics company called Surgisphere, raising major questions about the integrity of the studies’ findings.

The Lancet said it had issued the notice about the chloroquine study “to alert readers to the fact that serious scientific questions have been brought to our attention”6 after many researchers raised concerns about the methods and data integrity of the study.7

The NEJM said it had taken the action because “substantive concerns have been raised about the quality of the information” in the international database that its study drew on.8

Meanwhile, also on 2 June, the Annals of Internal Medicine retracted9 a paper10 published in April that concluded that surgical and cotton masks seemed to be ineffective in preventing the SARS-CoV-2 virus from spreading from the coughs of infected patients to the environment and the external surfaces of masks. In a statement published by the journal the authors conceded that they failed to recognise that the study of only four participants was below the limit of detection (LOD), rendering the findings “uninterpretable.”

This was highlighted after the study had been published and widely cited, including by WHO.11 The authors proposed correcting the reported data by adding extra patients, but the journal requested a retraction.

The authors’ statement said, “We had not fully recognized the concept of limit of detection (LOD) of the in-house reverse transcriptase polymerase chain reaction used in the study (2.63 log copies/mL), and we regret our failure to express the values below LOD as “<LOD (value).” The LOD is a statistical measure of the lowest quantity of the analyte that can be distinguished from the absence of that analyte. Therefore, values below the LOD are unreliable and our findings are uninterpretable.”

Marcus Munafò, professor of biological psychology at Bristol University’s MRC Integrative Epidemiology Unit, who leads the UK Reproducibility Network, said that the cases highlighted that the quest for fast answers shouldn’t override the need for quality.

He said, “These papers demonstrate how the pressing need for knowledge about measures we can take to counteract the covid-19 pandemic shouldn’t take priority over the need for the generation of that knowledge and its dissemination to be high quality. If we rush this work through, we’re more likely to make mistakes along the way, and that’s likely to be counterproductive. More haste, less speed—or at least, slower progress towards a solid evidence base.”