Research on covid-19 is suffering “imperfect incentives at every stage”BMJ 2020; 369 doi: https://doi.org/10.1136/bmj.m2045 (Published 28 May 2020) Cite this as: BMJ 2020;369:m2045
All rapid responses
Who will guard the guards themselves? COVID-19 research may be incomplete, but the role of the experts is no lesser during this pandemic
I concur with the author's concerns about research integrity and the peer review process.
Using the drug Hydroxychloroquine (HCQ) and its potential role in COVID-19 as an example, it is instrumental in various controversies during this pandemic:
1. Many HCQ studies on COVID-19 patient are published as "preprint" without peer review; websites like medRxiv seemed to thrive in their role as open access preprint clearing house. Despite the headings on each preprint article stating it "was not certified by peer review", many readers, including clinicians who should know better, do not place the signfiicance of this warning as much as they should and I suspect the definition and importance of what "peer review" plays in scientific literature.
2. Social media comments, including those from celebrities and world leaders, do not help in tempering the unfounded enthusiasm on the therapeutic possibilities of HCQ in COVID-19 patients. Often these comments are not properly referenced nor caveats included, leading to various assumption of other readers seeking the origins of sources of these information, and this adds another layer of mistaken authenticity and tick of approval by association.
3. Some medical authorities put out recommendations of treatment of COVID-19 patients and prophylaxis for healthcare workers; the Indian Council of Medical Research (ref 1) was accused of relying on "weak evidence to justify its recommendation". The specific prescriptive dosing of HCQ stated in its advisory in March 2020 adds another dimension on confidence in its recommendation of HCQ usage based on in vitro studies. Amazingly the same council resisted calls for further research on its own recommendation rejecting calls for proper trials to be done on people using HCQ based on the advisory.
4. There are concerns about the peer review process even for those articles published in prestigious medical journals. The current controversy involving the multinational registry analysis published in the Lancet attracted "methodological and data integrity concerns" from 146 international researchers (ref 2) who have put their opinion in an open letter. They stated that "In the interests of transparency, we also ask The Lancet to make openly available the peer review comments that led to this manuscript to be accepted for publication". (ref 3)
5. The World Health Organisation (WHO) halted its HCQ arm of Solidarity Trial based on the findings of the Lancet paper (ref 4). Its Director General stated that:
"The executive group has implemented a temporary pause of the hydroxychloroquine arm within the Solidarity Trial while the data is reviewed by the Data Safety Monitoring Board.” He added that, alongside the new research, the reviewers would also look at data collected so far in the Solidarity Trial—“in particular, robust randomised available data, to adequately evaluate the potential benefits and harms from this drug."
The HCQ arm of the Solidarity Trial has since resumed today (4 June 2020) (ref 5).
What is concerning is the revelation that for WHO-supervised trial under way for more than 2 months:
i) the WHO has not had adequate resources to maintain confidence and safety supervision of its HCQ arm over these 2 months such that it requires an immediate suspension within 2 days of the publication of the Lancet paper on 22 May 2020.
ii) that it took more than 10 days for it announce the continuation of the HCQ arm via media outlet, the decision was reported to have been taken on Friday (ref 5):
"On Friday, Adhanom Ghebreyesus said the World Health Organization had reviewed the Lancet study and other findings about hydroxychloroquine and had determined it was safe for its trials to continue".
That "Friday" would have been 29 May 2020 if the Guardian article (published on 4 June 2020) is to be believed.
This raises concerns about the adequacy and confidence of trial supervision and safety under the WHO, but also the resources available to WHO to deal with new information and decision making.
The BMJ is not untouched by the COVID-19 reporting problems either.
On 6 April 2020, the BMJ published an article headed "Covid-19: NICE advises against using NSAIDs for fever in patients with suspected cases" (ref 6), while the main article states the National Institute for Health and Care Excellence (NICE) advised "Paracetamol should be used in preference to non-steroidal anti-inflammatory drugs for managing fever in patients with suspected covid-19 until more evidence is available on the safety of NSAIDs"
The actual statement in the NICE rapid guideline section 5.4 as of 4 April 2020 is as follows:
"Advise patients to take paracetamol if they have fever and other symptoms that antipyretics would help treat. Tell them to continue only while the symptoms of fever and the other symptoms are present. Until there is more evidence, paracetamol is preferred to non-steroidal anti-inflammatory drugs (NSAIDs) for patients with COVID-19 (see Central Alerting System: novel coronavirus - antiinflammatory medications)."
The guidelines would not have supported the BMJ article headlines' inference that NICE had advised AGAINST the use of non-steroidal anti-inflammatory drugs for fever.
Similarly I have reservations on BMJ's peer review processes on COVID-19 research papers in which some obvious omission of considerations by the authors requires clarification. Some of my concerns regarding these apparent yet unaddressed issues have been published in rapid responses (ref 7,8).
Another example is Argenziano et al's (ref 9) descriptive listing of Hospital course and outcomes of inpatients which, while interesting, does not offer context in absence of relationship to patients' state of condition upon emergency department presentation. Confusing parameters such as numbers “extubated at least once” do not help in understanding the results being presented. I wonder if a more vigorous review would have altered the article to make more connections of its outcome findings with admission condition.
While I agree there is a need for transparency as suggested by Professor Munafò "to help ensure the (decision making) process is seen as trustworthy" when these decisions are "informed by high quality evidence, even if it is imperfect or incomplete", I am concerned the researchers, the clinicians and the publishers, who should know better, are not coping with the process.
Quis custodiet ipsos custodes?
Competing interests: No competing interests
Something is better than nothing.
The Scientific publications during COVID-19 pandemic have exposed the soft underbelly of evidence based medicine. Robust evidence requires rigorous data collection and analysis which in turn needs significant investment of time. The fast spreading Corona virus didn’t offer the luxury of waiting for the perfect scientific paper.
Fighting a pandemic is analogous to fighting a war. As every General knows, Military strategies derived from simulated pre-war games don’t fully equip an army to deal with all situations and often decisions are taken on battle-front using incomplete information. So any additional clinical information is better than no information at all when fighting to overcome this pandemic.
Some of the early reports such as the adverse impact of ACE inhibitors on COVID-19 risk and mortality were proven to be false. On the other hand, some of the early papers did provide useful information such as the impact of cardiovascular co-morbidity on COVID-19 mortality. These reports were subsequently confirmed by good quality papers.
Scientific progress depends on verifying or disapproving a hypothesis. The early case reports, although imperfect, were instrumental in generating lot of scientific hypotheses.
Competing interests: No competing interests