Covid-19: Validity of key studies in doubt after leading journals issue expressions of concernBMJ 2020; 369 doi: https://doi.org/10.1136/bmj.m2224 (Published 04 June 2020) Cite this as: BMJ 2020;369:m2224
All rapid responses
Shortly after the news article by Iacobucci was published, The Lancet and The New England Journal of Medicine retracted their manuscripts written by Mehra and colleagues due to the limited veracity of their data sources (i.e., Surgisphere’s Surgical Outcomes Collaborative). This event is another pronounced example suggested by the BMJ editorial from Glasziou and colleagues, which claimed many COVID-19 studies during the pandemic are “poor in quality” and lead to “waste” in research.
I think this story offers two lessons. First, not only trials but also observational studies during the pandemic can be poor in quality. In the case of retracted manuscripts, the source of the data was found to be unreliable. With poor reliability and validity of the data, the quality of observational studies becomes questionable, and those retracted manuscripts are good examples of a poor-quality observational study.
Second, the pandemic may have lowered the scientific standards. Those retracted manuscripts can be considered as examples of “slip through the net”. This crisis is especially worrisome because two retracted manuscripts were published by The Lancet and The New England Journal of Medicine, two of the world’s most prestigious medical journals. A short piece by London and Kimmelman supports this view by raising concerns that “crisis situations demand exceptions to high standards for quality” and gives a warning about “pandemic research exceptionalism”.
Two retracted manuscripts may be one of many examples of poor-quality COVID-19 research that has been done and are expected to come out in the future. For example, as Iacobucci suggested, another study was retracted by the Annals of Internal Medicine, another prestigious medical journal, with a study sample size of only four. Also, before these spotlighted retractions, several manuscripts were either retracted or withdrawn from journals and preprint websites. A more detailed list of retracted COVID-19 manuscripts is available in the Retraction Watch database.
However, the quality of research should not be sacrificed due to the urgency of the situation, because currently health care professionals, policymakers, and patients are relying heavily on scientific evidence to make decisions. Therefore, we, as researchers, need to stay vigilant and pursue high-quality evidence even in these challenging times.
1. Iacobucci G. Covid-19: Validity of Key Studies in Doubt After Leading Journals Issue Expressions of Concern. BMJ. 2020 Jun 4;369:m2224. doi: 10.1136/bmj.m2224.
2. Piller C, Servick K. Two elite medical journals retract coronavirus papers over data integrity questions. Science. June 4th, 2020. doi:10.1126/science.abd1697
3. Glasziou PP, Sanders S, Hoffmann T. Waste in covid-19 Research. BMJ. 2020 May 12;369:m1847. doi: 10.1136/bmj.m1847.
4. Offord C. Concerns Build Over Surgisphere’s COVID-19 Dataset. The Scientist. June 2nd, 2020.
5. London AJ, Kimmelman J. Against Pandemic Research Exceptionalism. Science. 2020 May 1;368(6490):476-477. doi: 10.1126/science.abc1731
6. Bae S, Kim MC, Kim JY, et al. Effectiveness of surgical and cotton masks in blocking SARS-CoV-2: a controlled comparison in 4 patients. Ann Intern Med. 2020 Apr 6;M20-1342. doi: 10.7326/M20-1342
7. Retraction Watch Database. 2020. Available from: https://retractionwatch.com Accessed June 5th, 2020.
Competing interests: KH: Received the Maryland CERSI Scholar award from the Food and Drug Administration (grant # 5U01FD005946-04).