Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers
BMJ 2018; 363 doi: https://doi.org/10.1136/bmj.k4645 (Published 12 November 2018) Cite this as: BMJ 2018;363:k4645Linked opinion
Is 85% of health research really “wasted”?
Linked opinion
Funders and regulators are more important than journals in fixing the waste in research
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
We agree with Sharp and Curlewis: All health care students, including medical students, should learn to be critical readers and users of research ("users") [1,2]; few students will wish or need to be able to lead research ("do-ers"). We waste students' time and serve them poorly by assuming that teaching them all to be do-ers will make them good users.
Our experience is the reverse: teaching all health care students and workers to be proficient users of research is essential to them all, and is a great start for the few who will become do-ers. And ideally the do-ers should learn about good - and poor - research from critical appraisal, and ideally by doing at least one systematic review [3], and by working with experienced researchers, as in the STARSurg (STudent Audit and Research in Surgery) multicentre research model.
As pointed out by Sreeharan and Ahmed and colleagues, change will require reduced incentives - for students and academics - to publish what they refer to as "low level publications". Incentives to publish can undermine incentives to "research for the right reasons" (Doug Altman). Poor research is not simply a waste in itself; it also distracts and distorts our institutional, clinical, and publication systems for doing research for the right reasons.
References
1. Glasziou P, Burls A, Gilbert R. Evidence based medicine and the medical curriculum. BMJ. 2008 Sep 24;337:a1253
2. Albarqouni L, Hoffmann T, Straus S, et al. Core Competencies in Evidence-Based Practice for Health ProfessionalsConsensus Statement Based on a Systematic Review and Delphi Survey. JAMA Netw Open.2018;1(2):e180281. doi:10.1001/jamanetworkopen.2018.0281
2. Mahtani KR. All health researchers should begin their training by preparing at least one systematic review. J R Soc Med. 2016 Jul;109(7):264-8.
Competing interests: Prof Glasziou is co-Director of an EQUATOR Centre and Chair of the REWARD Alliance.
The commentary by Glasziou and Chalmers 1 and the response on medical student research 2 raise some important issues that need active management to prevent the impact of poor and wasted research on patient safety and medical training, not to mention the colossal drainage of societal funds in the region of $85 Billion 1. What is even more intriguing is that contrary to common belief, academic research appears to be even more culpable than research funded by Industry as indicated by the Trial Tracker established at the Oxford Centre for Evidence Based Medicine.
Research sponsored by the Pharmaceutical Industry has rightly received considerable scrutiny over the years and several improvements have been seen, especially in the area of trial registration and publication of negative trials. But many of the areas on which the Industry has received scrutiny also applies equally to all forms of research, including academic research. One could even argue that since Academic research is not under the stringent regulatory oversight of Industry research, the slippery slope towards wastage can be even greater. As we argued before 3, the reason Industry funded research has been shown to have more positive trial-based outcomes could be due to the extraordinary preparation that goes before embarking on large scale clinical trials not only because of the regulatory scrutiny but also paradoxically due to economic reasons to maximise the chances of the trail meeting its primary objective. These include preparatory preclinical work, ensuring that heavy investments into large scale clinical programs are only made after the demonstration of Proof of Concept in smaller trials with stringent GO/NO Go criteria and the statistical rigour that goes into power calculations and establishment of the primary end points.
The need to declare Conflicts of Interests has almost always been restricted to potential commercial conflicts. But is it time to recognise other personal and academic conflicts that could drive wasted research? The pressure to “publish or perish”, which was primarily a phenomenon seen in North America many decades ago, has now become a global phenomenon that not only applies to many academics in low-middle income countries but worryingly has encroached the arena of undergraduate medical students and postgraduate trainees. Would the drive for academic promotions and the inclusion of academics as co-authors in student driven research be considered as examples of such conflicts?
References:
1.Glasziou P & Chalmers I. Research waste is still a scandal – an essay by Paul Glasziou and Ian Chalmers. BMJ.2018; 363: k4645
2. Sharp EW & Curlewis K. Research waste is still a scandal – especially in medical students BMJ 2018; 363 doi: https://doi.org/10.1136/bmj.k4645
3. Qizilbash N, Rockhold F & Sreeharan N. Industry sponsored trials; an alternative view. BMJ 2002; 325 doi: https://doi.org/10.1136/bmj.325.7358.249
Competing interests: None to declare. But my views are influenced by my positions as Professor of Medicine, Sri Lanka (low-middle income country), and Senior Vice President and European Medical Director, GlaxoSmithKline R&D (Pharmaceutical Industry)
We read with great interest the recent article by Sharp and Curlewis (1), who provide a medical student’s perspective to Glasziou and Chalmers’ recent article addressing the topic of research waste (2). The authors discuss a recognised pressure amongst UK medical students to publish research ‘at all costs’. Research collaboratives such as STARSurg (STudent Audit and Research in Surgery) allow medical students to engage in robust and high-quality research, whilst gaining fundamental skills in study design and delivery. STARSurg uses a single corporate authorship model for all its publications, recognising all collaborators equally with PubMed citable co-authorship under group name: ‘STARSurg Collaborative’. This democratises the traditional publication model and flattens hierarchies within study delivery teams. We agree that the current Foundation Programme Application Service (FPAS) scoring system could be seen to encourage quantity rather than quality of publications; for example rewarding published letters and case studies with ‘points’, but leaving PubMed-citable collaborative authorship without recognition. (3).
STARSurg medical student collaborators have had fundamental roles in designing and delivering four national and international audits to date, including over 20,000 patients in 25 countries. Students have had leading roles in major studies across very diverse settings, for example GlobalSurg-2, investigating surgical site infection across eighty low-, middle- and high-income countries (4), and in complex randomised controlled trials, such as the West Midlands Research Collaborative (WMRC) ‘Dexamethasone reduces emesis after major surgery’ (DREAMS) trial (5). Increasingly, large publicly-funded studies with real potential for patient benefit are turning to single corporate authorship model to reflect the complexity of integrated nature of their delivery teams (2). Protocols for these studies undergo extensive internal and external pre-publication peer review, review by external grant committees, and scrutiny from multi-country audit or ethical approvals processes; Sharp and Curlewis therefore rightly highlight that collaborative research is not only a viable solution, but an invaluable method of avoiding research waste. Individual students may contribute several weeks to months of high intensive efforts to make meaningful contributions to these studies, and gain the status as a citable collaborating author.
We are saddened therefore that the UK Foundation Programme Office (UKFPO) maintains its firm stance on recognition of collaborative research within the FPAS scoring system, despite multiple attempts at discourse; an open letter to the UKFPO on behalf of over 900 signatories has failed to make headway (6). We strongly urge the UKFPO to update its position on appropriate recognition of collaborative research, as is seen in the academic foundation programme, both core and higher specialist training applications, Annual Reviews of Competence Progression (ARCP), and an increasing number of specialities for Certificate of Completion of Training (CCT). Modification of the FPAS scoring system has a crucial role to play in promoting a culture of engagement with multi-centre research, reducing research waste caused by falsely inflating the importance of low-level publications, and increasing the likelihood of patient benefit.
References:
1. Sharp E, Curlewis K. Research waste is still a scandal—especially in medical students BMJ 2018; 363: k4645.
2. Paul G, Chalmers, I. Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers BMJ 2018; 363: k4645.
3. UK Foundation Programme Office (UKFPO). UK Foundation Programme 2019 Applicants’ Handbook. Available at: http://www.foundationprogramme.nhs.uk.
4. GlobalSurg Collaborative. Surgical site infection after gastrointestinal surgery in high-income, middle-income, and low-income countries: a prospective, international, multicentre cohort study. Lancet Infect Dis 2018; 18: 516–25.
5. DREAMS Trial Collaborators and West Midlands Research Collaborative. Dexamethasone versus standard treatment for postoperative nausea and vomiting in gastrointestinal surgery: randomised controlled trial (DREAMS Trial). BMJ. 2017; 357: j1455.
6. STARSurg Collaborative. Students’ participation in collaborative research should be recognised. Int J Surg. 2017; 39: 234 –7.
Authors:
Ahmed WUR* (University of Exeter Medical School, Exeter, UK, @WaheedURAhmed1)
Mills E (Brighton and Sussex Medical School, Brighton, UK, @Milye95)
Khaw RA (University of Manchester Medical School, Manchester, UK, @RachelAlys)
McLean KA (Academic Foundation Year 2, University of Edinburgh, UK, @Kenneth.McLean92).
Glasbey JC (Academic Clinical Fellow, University of Birmingham, UK, @DrJamesGlasbey).
*Correspondences to: collaborate@starsurg.org
Competing interests: No competing interests
Competing interests: No competing interests
We thank Paul Glasziou and Iain Chalmers for their informative essay, “Research waste is still a scandal” (1).
As medical students in the UK we already feel pushed to complete research. No group is more susceptible to designing, conducting and reporting poor quality research than medical students who have not had time to develop the essential skills needed to conduct meaningful research. A 2016 study of medical students in the USA found that only 23% of students pursued research for academic interest, with the largest group of students (32%) pursuing research to increase the competitiveness of their postgraduate job application (2). Some of the same ulterior motivators that encourage wasteful research in the USA also exist in the UK.
We feel it is necessary to complete research to remain competitive for future job applications as do many of our peers – similar to the study in the USA. The UK correctly focusses on “putting patients first” but houses systematic motivators to pursue research for reasons ulterior to those of improving patient care. For example, the UK uses standardised scoring systems to recruit junior doctors immediately after leaving medical school where authorship on up to two publications is rewarded by more points (3). These motivators often favour quantity over quality and exist in core training, speciality training and at consultancy (4). To highlight the pressure students’ face to publish research it is joked that, “soon you will have to be a published author to even get into medical school”.
Bernard M Y Cheung, editor of the BMJ Postgraduate Medical Journal, recently addressed the question of, “is [medical student research] necessary and beneficial?” (5). He highlights that is it not unusual for students to finish university having completed a research project. Observations like this increase the pressure we burden ourselves with to conduct research outside of our taught curriculum despite knowing the impact the project has will be small.
The challenge remains of how to encourage students and junior colleagues to engage with meaningful research, whilst not pursing a culture of quantity over quality. We have two suggestions to address this.
Firstly, we suggest a greater reward being placed on participation in well-designed and well-conducted research even if this is a small role in a large project that does not result in authorship. This will reduce the “85% of wasted effort in medical research” that Glasziou and Chalmers highlighted (1) by removing some of the burden on junior colleagues to pursue low impact research in order to have authorship on the publication. The teamwork involved in this will foster an environment where inexperienced members can develop advanced research skills without having to publish potentially wasteful research themselves. Unfortunately, collaborative research networks such as the Student Audit and Research in Surgery Collaborative (STARsurg) (6) do not reward participants with a PubMed ID and therefore students are not rewarded with UK Foundation Programme points for participating. We feel greater rewards should be given to those involved in these collaborative networks as they begin to teach students the important skills required conduct meaningful research.
This leads on to our second suggestion. Greater rewards must be given to those who demonstrate advanced knowledge and understanding of research methodology and critical appraisal. Whether you are an academic, clinician or manager, it is essential that you are able to recognise the quality of research you are reviewing and use the results produced accordingly. These skills are often assessed within academic foundation programme applications, yet not routinely assessed in normal UK Foundation Programme applications (7). We suggest that the current point system could be altered to include assessment of knowledge of research methodology and critical appraisal above and beyond what is expected for final exams. We feel these suggestions can begin to correct the issue of wastefulness that persists in medical research today.
We would like to hear the opinions of Paul Glasziou and Iain Chalmers on medical student research.
References:
1. Glasziou P, Chalmers I. Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers. BMJ. 2018;363:k4645.
2. Pathipati A, Taleghani N. Research in Medical School: A Survey Evaluating Why Medical Students Take Research Years. Cureus. 2016;8(8):e741.
3. UK Foundation Programme Office. UKFP 2019 Applicants' Handbook. London: UK Foundation Programme Office; 2018.
4. Health Education England. Person specifications [Internet]. Specialty Training. 2018 [cited 18 November 2018]. Available from: https://specialtytraining.hee.nhs.uk/Recruitment/Person-specifications
5. Cheung B. Medical student research: is it necessary and beneficial?. Postgraduate Medical Journal. 2018;94(1112):317-317.
6. STARsurg. Home [Internet]. STARsurg. 2018 [cited 19 November 2018]. Available from: https://starsurg.org/
7. UK Foundation Programme Office. Rough Guide to the Academic Foundation Programme. London: UK Foundation Programme Office; 2013. Available from: http://www.foundationprogramme.nhs.uk/sites/default/files/2018-08/Rough%...
Competing interests: No competing interests
Re: Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers
There is no argument with the reasons for rigour in research; the need to fulfil requirements and follow proper procedures is not in doubt. Where this rigour is not available is in the occasional chance finding; when an outcome is so unexpected that the normal preliminaries could not have been followed because the actual result was considered impossible. Clearly in this circumstance one cannot go back in time to provide the basic information required if the outcome is to be reported for others to research in more detail.
It can be argued that single serendipitous findings are rare, and so they are, but if they cannot be reported because of lack of rigour and process then progress will be slowed or not made at all.
Two examples:
1. A recent case of regression of symptoms of vascular dementia with Alzheimer's and return to near-normal cognition requires strong prior assessments and diagnostic procedures to avoid suggestions of incorrect diagnosis as a reason for the event. These cannot be done retrospectively but are not done routinely in the NHS where the diagnosis was made. Yet the outcome is worthy of report for follow-up by someone with foresight and funds.
2. A case of rapid resolution of trigeminal neuralgia without drug intervention relied on an accurate diagnosis in the first instance but it cannot be returned to after resolution has occurred. Fortunately this has now been taken to clinical trial in India.
It will be a shame if unachievable standards stop publication and potential follow-up of such events.
Competing interests: No competing interests