Intended for healthcare professionals

CCBY Open access
Research Methods & Reporting

Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness

BMJ 2020; 368 doi: https://doi.org/10.1136/bmj.l6927 (Published 20 March 2020) Cite this as: BMJ 2020;368:l6927
  1. Sebastian Vollmer, associate professor and health programme codirector1 2,
  2. Bilal A Mateen, research fellow1 3 4,
  3. Gergo Bohner, postdoctoral research fellow1 2,
  4. Franz J Király, lecturer1 5,
  5. Rayid Ghani, distinguished career professor6,
  6. Pall Jonsson, associate director7,
  7. Sarah Cumbers, associate director8,
  8. Adrian Jonas, associate director9,
  9. Katherine S L McAllister, technical adviser9,
  10. Puja Myles, head of observational research10,
  11. David Grainger, device specialist11,
  12. Mark Birse, head of inspectorate and process licencing11,
  13. Richard Branson, senior manager11,
  14. Karel G M Moons, professor of epidemiology12,
  15. Gary S Collins, professor of medical statistics13,
  16. John P A Ioannidis, professor of medicine14,
  17. Chris Holmes, professor of biostatistics and health programme scientific director1 15,
  18. Harry Hemingway, professor of clinical epidemiology and research director16 17 18
  1. 1Alan Turing Institute, Kings Cross, London, UK
  2. 2Departments of Mathematics and Statistics, University of Warwick, Coventry, UK
  3. 3Warwick Medical School, University of Warwick, Coventry, UK
  4. 4Kings College Hospital, Denmark Hill, London, UK
  5. 5Department of Statistical Science, University College London, London, UK
  6. 6University of Chicago, Chicago, IL, USA
  7. 7Science Policy and Research, National Institute for Health and Care Excellence, Manchester, UK
  8. 8Health and Social Care Directorate, National Institute for Health and Care Excellence, London, UK
  9. 9Data and Analytics Group, National Institute for Health and Care Excellence, London, UK
  10. 10Clinical Practice Research Datalink, Medicines and Healthcare products Regulatory Agency, London, UK
  11. 11Medicines and Healthcare products Regulatory Agency, London, UK
  12. 12Julius Centre for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, Utrecht, Netherlands
  13. 13UK EQUATOR Centre, Centre for Statistics in Medicine, NDORMS, University of Oxford, Oxford, UK
  14. 14Meta-Research Innovation Centre at Stanford, Stanford University, Stanford, CA, USA
  15. 15Department of Statistics, University of Oxford, Oxford OX1 3LB, UK
  16. 16Health Data Research UK London, University College London, London, UK
  17. 17Institute of Health Informatics, University College London, London, UK
  18. 18National Institute for Health Research, University College London Hospitals Biomedical Research Centre, University College London, London, UK
  1. Correspondence to: C Holmes cholmes{at}stats.ox.ac.uk
  • Accepted 22 October 2019

Machine learning, artificial intelligence, and other modern statistical methods are providing new opportunities to operationalise previously untapped and rapidly growing sources of data for patient benefit. Despite much promising research currently being undertaken, particularly in imaging, the literature as a whole lacks transparency, clear reporting to facilitate replicability, exploration for potential ethical concerns, and clear demonstrations of effectiveness. Among the many reasons why these problems exist, one of the most important (for which we provide a preliminary solution here) is the current lack of best practice guidance specific to machine learning and artificial intelligence. However, we believe that interdisciplinary groups pursuing research and impact projects involving machine learning and artificial intelligence for health would benefit from explicitly addressing a series of questions concerning transparency, reproducibility, ethics, and effectiveness (TREE). The 20 critical questions proposed here provide a framework for research groups to inform the design, conduct, and reporting; for editors and peer reviewers to evaluate contributions to the literature; and for patients, clinicians and policy makers to critically appraise where new findings may deliver patient benefit.

Machine learning (ML), artificial intelligence (AI), and other modern statistical methods are providing new opportunities to operationalise previously untapped and rapidly growing sources of data for patient benefit. The potential uses include improving diagnostic accuracy,1 more reliably predicting prognosis,2 targeting treatments,3 and increasing the operational efficiency of health systems.4 Examples of potentially disruptive technology with early promise include image based diagnostic applications of ML/AI, which have shown the most early clinical promise (eg, deep learning based algorithms improving accuracy in diagnosing retinal pathology compared with that of specialist physicians5), or natural language processing used as a tool to extract information from structured and unstructured (that is, free) text embedded in electronic health records.2 Although we are only just …

View Full Text