Intended for healthcare professionals

  1. Research
  2. Reporting guideline...
  3. Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI
CCBYNC Open access
Research Methods & Reporting

Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

BMJ 2022; 377 doi: https://doi.org/10.1136/bmj-2022-070904 (Published 18 May 2022) Cite this as: BMJ 2022;377:e070904
  1. Baptiste Vasey, DPhil candidate1 2 3,
  2. Myura Nagendran, academic clinical fellow4,
  3. Bruce Campbell, professor5 6,
  4. David A Clifton, professor2,
  5. Gary S Collins, professor7,
  6. Spiros Denaxas, professor8 9 10 11,
  7. Alastair K Denniston, professor12 13 14,
  8. Livia Faes, clinical fellow14,
  9. Bart Geerts, chief executive officer15,
  10. Mudathir Ibrahim, research fellow1 16,
  11. Xiaoxuan Liu, clinical research fellow12 13,
  12. Bilal A Mateen, deputy director8 17 18,
  13. Piyush Mathur, staff anaesthesiologist19,
  14. Melissa D McCradden, assistant professor20 21,
  15. Lauren Morgan, human factors specialist22,
  16. Johan Ordish, deputy director23,
  17. Campbell Rogers, chief medical officer24,
  18. Suchi Saria, chief executive officer25 26,
  19. Daniel S W Ting, associate professor27 28,
  20. Peter Watkinson, professor3 29,
  21. Wim Weber, associate professor30,
  22. Peter Wheatstone, patient representative31,
  23. Peter McCulloch, professor1
  24. on behalf of the DECIDE-AI expert group
    1. 1Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK
    2. 2Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK
    3. 3Critical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
    4. 4UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK
    5. 5University of Exeter Medical School, Exeter, UK
    6. 6Royal Devon and Exeter Hospital, Exeter, UK
    7. 7Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
    8. 8Institute of Health Informatics, University College London, London, UK
    9. 9British Heart Foundation Data Science Centre, London, UK
    10. 10Health Data Research UK, London, UK
    11. 11UCL Hospitals Biomedical Research Centre, London, UK
    12. 12University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
    13. 13Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
    14. 14Moorfields Eye Hospital NHS Foundation Trust, London, UK
    15. 15Healthplus.ai-R&D, Amsterdam, Netherlands
    16. 16Department of Surgery, Maimonides Medical Center, New York, NY, USA
    17. 17Wellcome Trust, London, UK
    18. 18Alan Turing Institute, London, UK
    19. 19Department of General Anesthesiology, Anesthesiology Institute, Cleveland Clinic, Cleveland, OH, USA
    20. 20Hospital for Sick Children, Toronto, ON, Canada
    21. 21Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada
    22. 22Morgan Human Systems, Shrewsbury, UK
    23. 23The Medicines and Healthcare products Regulatory Agency, London, UK
    24. 24HeartFlow, Redwood City, CA, USA
    25. 25Departments of Computer Science, Statistics, and Health Policy, and Division of Informatics, Johns Hopkins University, Baltimore, MD, USA
    26. 26Bayesian Health, New York, NY, USA
    27. 27Singapore National Eye Center, Singapore Eye Research Institute, Singapore
    28. 28Duke-NUS Medical School, National University of Singapore, Singapore
    29. 29NIHR Biomedical Research Centre Oxford, Oxford University Hospitals NHS Trust, Oxford, UK
    30. 30The BMJ, London, UK
    31. 31School of Medicine, University of Leeds, Leeds, UK
    1. Correspondence to: B Vasey Nuffield Department of Surgical Sciences, University of Oxford, Oxford OX3 9DU, UK baptiste.vasey{at}gmail.com
    • Accepted 26 April 2022

    A growing number of artificial intelligence (AI)-based clinical decision support systems are showing promising performance in preclinical, in silico, evaluation, but few have yet demonstrated real benefit to patient care. Early stage clinical evaluation is important to assess an AI system’s actual clinical performance at small scale, ensure its safety, evaluate the human factors surrounding its use, and pave the way to further large scale trials. However, the reporting of these early studies remains inadequate. The present statement provides a multistakeholder, consensus-based reporting guideline for the Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence (DECIDE-AI). We conducted a two round, modified Delphi process to collect and analyse expert opinion on the reporting of early clinical evaluation of AI systems. Experts were recruited from 20 predefined stakeholder categories. The final composition and wording of the guideline was determined at a virtual consensus meeting. The checklist and the Explanation & Elaboration (E&E) sections were refined based on feedback from a qualitative evaluation process. 123 experts participated in the first round of Delphi, 138 in the second, 16 in the consensus meeting, and 16 in the qualitative evaluation. The DECIDE-AI reporting guideline comprises 17 AI specific reporting items (made of 28 subitems) and 10 generic reporting items, with an E&E paragraph provided for each. Through consultation and consensus with a range of stakeholders, we have developed a guideline comprising key items that should be reported in early stage clinical studies of AI-based decision support systems in healthcare. By providing an actionable checklist of minimal reporting items, the DECIDE-AI guideline will facilitate the appraisal of these studies and replicability of their findings.

    The prospect of improved clinical outcomes and more efficient health systems has fuelled a rapid rise in the development and evaluation of artificial intelligence (AI) systems over the last decade. Because most AI systems within healthcare are complex interventions designed as clinical decision support systems, rather than autonomous agents, the interactions between the AI systems, their users and the implementation environments are defining components of the AI interventions’ overall potential effectiveness. Therefore, bringing AI systems from mathematical performance to clinical utility, needs an adapted, stepwise implementation and evaluation pathway, addressing the complexity of this collaboration between two independent forms of intelligence, beyond measures of effectiveness alone.1 Despite indications that some AI-based algorithms now match the accuracy of human experts within preclinical in silico studies,2 there is little high quality evidence for improved clinician performance or patient outcomes in clinical studies.34 Reasons proposed for this so called AI-chasm5 are lack of necessary expertise needed for translating a tool into practice, lack of funding available for translation, a general underappreciation of clinical research as a translation mechanism6 and more specifically a disregard for the potential value of the early stages of clinical evaluation and the analysis of human factors.7

    Key messages

    • DECIDE-AI is a stage specific reporting guideline for the early, small scale and live clinical evaluation of decision support systems based on artificial intelligence

    • The DECIDE-AI checklist presents 27 items considered as minimum reporting standards. It is the result of a consensus process involving 151 experts from 18 countries and 20 stakeholder groups

    • DECIDE-AI aims to improve the reporting around four key aspects of early stage live AI evaluation: proof of clinical utility at small scale, safety, human factors evaluation, and preparation for larger scale summative trials

    The challenges of early stage clinical AI evaluation (see box 1) are similar to those of complex interventions, as reported by the Medical Research Council dedicated guidance,1 and surgical innovation, as described by the IDEAL Framework.89 For example, in all three cases, the evaluation needs to consider the potential for iterative modification of the interventions and the characteristics of the operators (or users) performing them. In this regard, the IDEAL framework offers readily implementable and stage specific recommendations for the evaluation of surgical innovations under development. IDEAL stages 2a/2b, for example, are described as development and exploratory stages, during which the intervention is refined, operators’ learning curves analysed, and the influence of patient and operator variability on effectiveness are explored prospectively, prior to large scale efficacy testing.

    Box 1

    Methodological challenges of the artificial intelligence (AI)-based decision support system evaluation

    • The clinical evaluation of AI-based decision support systems presents several methodological challenges, all of which will likely be encountered at early stage. These are the needs to:

      • account for the complex intervention nature of these systems and evaluate their integration within existing ecosystems

      • account for user variability and the added biases occurring as a result

      • consider two collaborating forms of intelligence (human and AI system) and therefore integrate human factors considerations as a core component

      • consider both physical patients and their data representations

      • account for the changing nature of the intervention (either due to early prototyping, version updates, or continuous learning design) and to analyse related performance changes

      • minimise the potential of this technology to embed and reproduce existing health inequality and systemic biases

      • estimate the generalisability of findings across sites and populations

      • enable reproducibility of the findings in the context of a dynamic innovation field and intellectual property protection

    RETURN TO TEXT

    Early stage clinical evaluation of AI systems should also place a strong emphasis on validation of performance and safety, in a similar manner to phase 1 and 2 pharmaceutical trials, before efficacy evaluation at scale in phase 3. For example, small changes in the distribution of the underlying data between the algorithm training and clinical evaluation populations (so called dataset shift) can lead to significant variation in clinical performance and expose patients to potential unexpected harm.1011

    Human factors (or ergonomics) evaluations are commonly conducted in safety critical fields such as aviation, the military and energy sectors.121314 Their assessments evaluate the impact of a device or procedure on their users’ physical and cognitive performance, and vice versa. Human factors, such as usability evaluation, are an integral part of the regulatory process for new medical devices1516 and their application to AI specific challenges is attracting growing attention in the medical literature.17181920 However, few clinical AI studies report on the evaluation of human factors,3 and usability evaluation of related digital health technology is often performed with inconstant methodology and reporting.21

    Other areas of suboptimal reporting of clinical AI studies have also recently been highlighted,322 such as implementation environment, user characteristics and selection process, training provided, underlying algorithm identification, and disclosure of funding sources. Transparent reporting is necessary for informed study appraisal and to facilitate reproducibility of study results. In a relatively new and dynamic field such as clinical AI, comprehensive reporting is also key to construct a common and comparable knowledge base to build upon.

    Guidelines already exist, or are under development, for the reporting of preclinical, in silico, studies of AI systems, their offline validation, and for their evaluation in large comparative studies23242526; but there is an important stage of research between these, namely studies focussing on the initial clinical use of AI systems, for which no such guidance currently exists (fig 1 and table 1). This early clinical evaluation provides a crucial scoping evaluation of clinical utility, safety, and human factors challenges in live clinical settings. By investigating the potential obstacles to clinical evaluation at scale and informing protocol design, these studies are also important stepping stones toward definitive comparative trials.

    Fig 1
    Fig 1

    Comparison of development pathways for drug therapies, artificial intelligence (AI) in healthcare, and surgical innovation. The coloured lines represent reporting guidelines, some of which are study design specific (TRIPOD-AI, STARD-AI, SPIRIT/CONSORT, SPIRIT/CONSORT-AI), others stage specific (DECIDE-AI, IDEAL). Depending on the context, more than one study design can be appropriate for each stage. *Only apply to AI in healthcare

    Table 1

    Overview of existing and upcoming artificial intelligence (AI) reporting guidelines

    View this table:

    To address this gap, we convened an international, multistakeholder group of experts in a Delphi exercise to produce the DECIDE-AI reporting guideline. Focusing on AI systems supporting, rather than replacing human intelligence, DECIDE-AI aims to improve the reporting of studies describing the evaluation of AI-based decision support systems during their early, small scale implementation in live clinical settings (ie, the supported decisions have an actual impact on patient care). Whereas TRIPOD-AI, STARD-AI, SPIRIT-AI, and CONSORT-AI are specific to particular study designs, DECIDE-AI is focused on the evaluation stage and does not prescribe a fixed study design.

    Methods

    The DECIDE-AI guideline was developed through an international expert consensus process and in accordance with the EQUATOR Network’s recommendations for guideline development.27 A Steering Group was convened to oversee the guideline development process. Its members were selected to cover a broad range of expertise and ensure a seamless integration with other existing guidelines. We conducted a modified Delphi process,28 with two rounds of feedback from participating experts and one virtual consensus meeting. The project was reviewed by the University of Oxford Central University Research Ethics Committee (approval number R73712/RE003) and registered with the EQUATOR Network. Informed consent was obtained from all participants in the Delphi process and consensus meeting.

    Initial item list generation

    An initial list of candidate items was developed based on expert opinion informed by a systematic literature review focusing on the evaluation of AI-based diagnostic decision support systems,3 an additional literature search about existing guidance for AI evaluation in clinical settings (search strategy available on the Open Science Framework29), literature recommended by Steering Group members,19223031323334 and institutional documents.35363738

    Expert recruitment

    Experts were recruited through five different channels: invitation to experts recommended by the Steering Group, invitation to authors of the publications identified through the initial literature searches, call to contribute published in a commentary article in a medical journal,7 consideration of any expert contacting the Steering Group of their own initiative, and invitation to experts recommended by the Delphi participants (snowballing). Before starting the recruitment process, 20 target stakeholder groups were defined, namely: administrators/hospital management, allied health professionals, clinicians, engineers/computer scientists, entrepreneurs, epidemiologists, ethicists, funders, human factors specialists, implementation scientists, journal editors, methodologists, patient representatives, payers/commissioners, policy makers/official institution representatives, private sector representatives, psychologists, regulators, statisticians, and trialists.

    One hundred and thirty eight experts agreed to participate in the first round of Delphi, of whom 123 (89%) completed the questionnaire (83 identified from Steering Group recommendation, 12 from their publications, 21 contacting the Steering Group from of own initiative, and seven through snowballing). One hundred and sixty two experts were invited to take part in the second round of Delphi, of whom 138 completed the questionnaire (85%). 110 had also completed the first round (continuity rate of 89%)39 and 28 were new participants. The participating experts represented 18 countries and spanned all 20 of the defined stakeholder groups (see supplementary notes 1 and supplementary tables 1 and 2).

    Delphi process

    The Delphi surveys were designed and distributed via the REDCap web application.4041 The first round consisted of four open ended questions on aspects viewed by the Delphi participants as necessary to be reported during early stage clinical evaluation. The participating experts were then asked to rate, on a 1 to 9 scale, the importance of items in the initial list proposed by the research team. Ratings of 1 to 3 on the scale were defined as “not important,” 4 to 6 as “important but not critical,” and 7 to 9 as “important and critical.” Participants were also invited to comment on existing items and to suggest new items. An inductive thematic analysis of the narrative answers was performed independently by two reviewers (BV and MN) and conflict was resolved by consensus.42 The themes identified were used to correct any omissions in the initial list and to complement the background information about proposed items. Summary statistics of the item scores were produced for each stakeholder group, by calculating the median score, interquartile range, and the percentage of participants scoring an item 7 or higher, as well as 3 or lower, which were the prespecified inclusion and exclusion cut-offs, respectively). A revised item list was developed based on the results of the first round.

    In the second round, the participants were shown the results of the first round and invited to rate and comment on the items in the revised list. The detailed survey questions of the two rounds of Delphi can be found on the Open Science Framework (OSF).29 All analyses of item scores and comments were performed independently by two members of the research team (BV and MN), using NVivo (QSR International, v1.0) and Python (Python Software Foundation, v3.8.5). Conflicts were resolved by consensus.

    The initial item list contained 54 items. 120 sets of responses were included in the analysis of the first round of Delphi (one set of responses was excluded due to a reasonable suspicion of scale inversion, two due to completion after the deadline). The first round yielded 43 986 words of free text answers to the four initial open ended questions, 6,419 item scores, 228 comments, and 64 proposals for new items. The thematic analysis identified 109 themes. In the revised list, nine items remained unchanged, 22 were reworded/completed, 21 reorganised (merged/split, becoming 13 items), two items dropped, and nine new items added, for a total of 53 items. The two items dropped were related to health economic assessment. They were the only two items with a median score below 7 (median 6, interquartile range 2-9 for both) and received numerous comments describing them as an entirely separate aspect of evaluation. The revised list was reorganised into items and subitems. 136 sets of answers were included in the analysis of the second round of Delphi (one set of answers was excluded due to lack of consideration for the questions, one due to completion after the deadline). The second round yielded 7101 item scores and 923 comments. The results of the thematic analysis, the initial and revised item lists, as well as per item narrative and graphical summaries of the feedback received in both rounds can be found on OSF.29

    Consensus meeting

    A virtual consensus meeting was held on three occasions between the 14 and 16 of June 2021, to debate and agree the content and wording of the DECIDE-AI reporting guideline. The 16 members of the Consensus Group (see supplementary notes 1 and supplementary tables 2a and 2b) were selected to ensure a balanced representation of the key stakeholder groups, as well as geographic diversity. All items from the second round of Delphi were discussed and voted on during the consensus meeting. For each item, the results of the Delphi process were presented to the Consensus Group members and a vote was carried out anonymously using the Vevox online application (www.vevox.com). A prespecified cut-off of 80% of the Consensus Group members (excluding blank votes and abstentions) was necessary for an item to be included. To highlight the new, AI specific reporting items, the Consensus Group divided the guidelines into two item lists: an AI specific items list, which represents the main novelty of the DECIDE-AI guideline, and a second list of generic reporting items, which achieved high consensus but are not AI specific and could apply to most types of study. The Consensus Group selected 17 items (made of 28 subitems in total) for inclusion in the AI specific list and 10 items for inclusion in the generic reporting item list. Supplementary table 3 provides a summary of the Consensus Group meeting votes.

    Qualitative evaluation

    The drafts of the guideline and of the Explanation and Elaboration (E&E) sections were sent for qualitative evaluation to a group of 16 selected experts with experience in AI system implementation or in the peer reviewing of literature related to AI system evaluation (see supplementary notes 1), all of whom were independent of the Consensus Group. These 16 experts were asked to comment on the clarity and applicability of each AI specific item, using a custom form (available on OSF29). Item wording amendments and modifications to the E&E sections were conducted based on the feedback from the qualitative evaluation, which was independently analysed by two reviewers (BV and MN) and with conflicts resolved by consensus. A glossary of terms (see box 2) was produced to clarify key concepts used in the guideline. The Consensus Group approved the final item lists including any changes made during the qualitative evaluation. Supplementary figures 1 and 2 provide graphical representations of the two item lists’ (AI specific and generic) evolution.

    Box 2

    Glossary of terms

    AI system

    • Decision support system incorporating AI and consisting of: (i) the artificial intelligence or machine learning algorithm; (ii) the supporting software platform; and (iii) the supporting hardware platform

    AI system version

    • Unique reference for the form of the AI system and the state of its components at a single point in time. Allows for tracking changes to the AI system over time and comparing between different versions

    Algorithm

    • Mathematical model responsible for learning from data and producing an output

    Artificial intelligence (AI)

    • “Science of developing computer systems which can perform tasks normally requiring human intelligence”26

    Bias

    • “Systematic difference in treatment of certain objects, people, or groups in comparison to others”43

    Care pathway

    • Series of interactions, investigations, decision making and treatments experienced by patients in the course of their contact with a healthcare system for a defined reason

    Clinical

    • Relating to the observation and treatment of actual patients rather than in silico or scenario-based simulations

    Clinical evaluation

    • Set of ongoing activities, analysing clinical data and using scientific methods, to evaluate the clinical performance, effectiveness and/or safety of an AI system, when used as intended35

    Clinical investigation

    • Study performed on one or more human subjects to evaluate the clinical performance, effectiveness and/or safety of an AI system.44 This can be performed in any setting (eg, community, primary care, hospital)

    Clinical workflow

    • Series of tasks performed by healthcare professionals in the exercise of their clinical duties

    Decision support system

    • System designed to support human decision making by providing person specific and situation specific information or recommendations, to improve care or enhance health

    Exposure

    • State of being in contact with, and having used, an AI system or similar digital technology.

    Human-computer interaction

    • Bidirectional influence between human users and digital systems through a physical and conceptual interface

    Human factors

    • Also called ergonomics. “The scientific discipline concerned with the understanding of interactions among humans and other elements of a system, and the profession that applies theory, principles, data and methods to design in order to optimise human well-being and overall system performance.” (International Ergonomics Association)

    Indication for use

    • Situation and reason (medical condition, problem, and patient group) where the AI system should be used

    In silico evaluation

    • Evaluation performed via computer simulation outside the clinical settings

    Intended use

    • Use for which an AI system is intended, as stated by its developers, and which serves as the basis for its regulatory classification. The intended use includes aspects of: the targeted medical condition, patient population, user population, use environment, mode of action

    Learning curves

    • Graphical plotting of user performance against experience.45 By extension, analysis of the evolution of user performance with a task as exposure to the task increases. The measure of performance often uses other context specific metrics as a proxy

    Live evaluation

    • Evaluation under actual clinical conditions, in which the decisions made have a direct impact on patient care. As opposed to “offline” or “shadow mode” evaluation where the decisions do not have a direct impact on patient care

    Machine learning

    • “Field of computer science concerned with the development of models/algorithms that can solve specific tasks by learning patterns from data, rather than by following explicit rules. It is seen as an approach within the field of AI”26

    Participant

    • Subject of a research study, on which data will be collected and from whom consent is obtained (or waived). The DECIDE-AI guideline considers that both patients and users can be participants

    Patient

    • Person (or the digital representation of this person) receiving healthcare attention or using health services, and who is the subject of the decision made with the support of the AI system. NB: DECIDE-AI uses the term “patient” pragmatically to simplify the reading of the guideline. Strictly speaking, a person with no health conditions who is the subject of a decision made about them by an AI-based decision support tool to improve their health and wellbeing or for a preventative purpose is not necessarily a “patient” per se

    Patient involvement in research

    • Research carried out “with” or “by” patients or members of the public rather than “to”, “about” or “for” them. (Adapted from the INVOLVE definition of public involvement)

    Standard practice

    • Usual care currently received by the intended patient population for the targeted medical condition and problem. This may not necessarily be synonymous with the state-of-the-art practice

    Usability

    • “Extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use”46

    User

    • Person interacting with the AI system to inform their decision making. This person could be a healthcare professional or a patient

    • The definitions given pertain to the specific context of DECIDE-AI and the use of the terms in the guideline. They are not necessarily generally accepted definitions and might not always be fully applicable to other areas of research

    RETURN TO TEXT

    Recommendations

    Reporting item checklist

    The DECIDE-AI guideline should be used for the reporting of studies describing the early stage live clinical evaluation of AI-based decision support systems, independently of the study design chosen (fig 1 and table 1). Depending on the chosen study design and if available, authors may also wish to complete the reporting according to study type specific guideline (eg, STROBE for cohort studies).47Table 2 presents the DECIDE-AI checklist, comprising of the 17 AI specific reporting items and 10 generic reporting items selected by the Consensus Group. Each item comes with an E&E to explain why and how reporting is recommended (see supplementary appendix 1). A downloadable version of the checklist, designed to help researchers and reviewers check compliance when preparing or reviewing a manuscript, is available as supplementary appendix 2. Reporting guidelines are a set of minimum reporting recommendations and not intended to guide research conduct. Although familiarity with DECIDE-AI might be useful to inform some aspects of the design and conduct of studies within the guideline’s scope,48 adherence to the guideline alone should not be interpreted as an indication of methodological quality (which is the realm of methodological guidelines and risk of bias assessment tools). With increasingly complex AI interventions and evaluations, it might become challenging to report all the required information within a single primary manuscript, in which case references to the study protocol, open science repositories, related publications, and supplementary materials are encouraged.

    Table 2

    DECIDE-AI checklist

    View this table:

    Discussion

    The DECIDE-AI guideline is the result of an international consensus process involving a diverse group of experts spanning a wide range of professional background and experience. The level of interest across stakeholder groups and the high response rate amongst the invited experts speaks to the perceived need for more guidance in the reporting of studies presenting the development and evaluation of clinical AI systems, and to the growing value placed on comprehensive clinical evaluation to guide implementation. The emphasis placed on the role of human-in-the-loop decision making was guided by the Steering Group’s belief that AI will, at least in the foreseeable future, augment rather than replace human intelligence in clinical settings. In this context, thorough evaluation of the human-computer interaction and the roles played by the human users will be key to realising the full potential of AI.

    The DECIDE-AI guideline is the first stage specific AI reporting guideline to be developed. This stage specific approach echoes recognised development pathways for complex interventions,18949 and aligns conceptually with proposed frameworks for clinical AI,6505152 although no commonly agreed nomenclature or definition has so far been published for the stages of evaluation in this field. Given the current state of clinical AI evaluation, and the apparent deficit in reporting guidance for the early clinical stage, the DECIDE-AI Steering Group considered it important to crystallise current expert opinion into a consensus, to help improve reporting of these studies. Beside this primary objective, the DECIDE-AI guideline will hopefully also support authors during study design, protocol drafting and study registration, by providing them with clear criteria around which to plan their work. As with other reporting guidelines, it is important to note that the overall impact on the standard of reporting will need to be assessed in due course, once the wider community has had a chance to use the checklist and explanatory documents, which is likely to prompt modification and fine tuning of the DECIDE-AI guideline, based on its real world use. While the outcome of this process cannot be prejudged, there is evidence that the adoption of consensus-based reporting guidelines (such as CONSORT) does indeed improve the standard of reporting.53

    The Steering Group paid special attention to the integration of DECIDE-AI within the broader scheme of AI guidelines (eg, TRIPOD-AI, STARD-AI, SPIRIT-AI, and CONSORT-AI). It also focussed on DECIDE-AI being applicable to all type of decision support modalities (ie, detection, diagnostic, prognostic, and therapeutic). The final checklist should be considered as minimum scientific reporting standards and do not preclude reporting additional information, nor are they a substitute for other regulatory reporting or approval requirements. The overlap between scientific evaluation and regulatory processes was a core consideration during the development of the DECIDE-AI guideline. Early stage scientific studies can be used to inform regulatory decisions (eg, based on the stated intended use within the study), and are part of the clinical evidence generation process (eg, clinical investigations). The initial item list was aligned with information commonly required by regulatory agencies and regulatory considerations are introduced in the E&E paragraphs. However, given the somewhat different focuses of scientific evaluation and regulatory assessment,54 as well as differences between regulatory jurisdictions, it was decided to make no reference to specific regulatory processes in the guideline, nor to define the scope of DECIDE-AI within any particular regulatory framework. The primary focus of DECIDE-AI is scientific evaluation and reporting, for which regulatory documents often provide little guidance.

    Several topics led to more intense discussion than others, both during the Delphi process and Consensus Group discussion. Regardless of whether the corresponding items were included or not, these represent important issues that the AI and healthcare communities should consider and continue to debate. Firstly, we discussed at length whether users (see glossary of terms) should be considered as study participants. The consensus reached was that users are a key study population, about whom data will be collected (eg, reasons for variation from the AI system recommendation, user satisfaction, etc), who might logically be consented as study participants, and therefore should be considered as such. Because user characteristics (eg, experience) can affect intervention efficacy, both patient and user variability should be considered when evaluating AI systems, and reported adequately.

    Secondly, the relevance of comparator groups in early stage clinical evaluation was considered. Most studies retrieved in the literature search described a comparator group (commonly the same group of clinicians without AI support). Such comparators can provide useful information for the design of future large scale trials (eg, information on the potential effect size). However, comparator groups are often unnecessary at this early stage of clinical evaluation, when the focus is on issues other than comparative efficacy. Small scale clinical investigations are also usually underpowered to make statistically significant conclusions about efficacy, accounting for both patient and user variability. Moreover, the additional information gained from comparator groups in this context can often be inferred from other sources, like previous data on unassisted standard of care in the case of the expected effect size. Comparison groups are therefore mentioned in item VII but considered optional.

    Thirdly, output interpretability is often described as important to increase user and patient trust in the AI system, to contextualise the system’s outputs within the broader clinical information environment,19 and potentially for regulatory purpose.55 However, some experts argued that an output’s clinical value may be independent of its interpretability, and that the practical relevance of evaluating interpretability is still debatable.5657 Furthermore, there is currently no generally accepted way of quantifying or evaluating interpretability. For this reason, the Consensus Group decided not to include an item on interpretability at the current time.

    Fourthly, the notion of users’ trust in the AI system, and its evolution with time, were discussed. As users accumulate experience with, and receive feedback from, the real world use of AI systems, they will adapt their level of trust in its recommendations. Whether appropriate or not, this level of trust will influence, as recently demonstrated by McIntosh et al,58 how much impact the systems have on the final decision making and therefore influence the overall clinical performance of the AI system. Understanding how trust evolves is essential for planning user training and determining the optimal timepoints at which to start data collection in comparative trials. However, as for interpretability, there is currently no commonly accepted way to measure trust in the context of clinical AI. For this reason, the item about user trust in the AI system was not included in the final guideline. The fact that interpretability and trust were not included highlights the tendency of consensus-based guidelines development towards conservatism, because only widely agreed upon concepts reach the level of consensus needed for inclusion. However, changes of focus in the field as well as new methodological development can be integrated into subsequent guideline iterations. From this perspective, the issues of interpretability and trust are far from irrelevant to future AI evaluations and their exclusion from the current guideline reflects less a lack of interest than a need for further research into how we can best operationalise these metrics for the purposes of evaluation in AI systems.

    Fifthly, the notion of modifying the AI system (the intervention) during the evaluation received mixed opinions. During comparative trials, changes made to the intervention during data collection are questionable unless the changes are part of the study protocol; some authors even consider them as impermissible, on the basis that they would make valid interpretation of study results difficult or impossible. However, the objectives of early clinical evaluation are often not to make definitive conclusions on effectiveness. Iterative design evaluation cycles, if performed safely and reported transparently, offer opportunities to tailor an intervention to its users and beneficiaries, and augment chances of adoption of an optimised, fixed version during later summative evaluation.895960

    Sixthly, several experts noted the benefit of conducting human factors evaluation prior to clinical implementation and considered that therefore human factors should be reported separately. However, even robust preclinical human factors evaluation will not reliably characterise all the potential human factors issues which might arise during the use of an AI system in a live clinical environment, warranting a continued human factors evaluation at the early stage of clinical implementation. The Consensus Group agreed that human factors play a fundamental role in AI system adoption in clinical settings at scale and that the full appraisal of an AI system’s clinical utility can only happen in the context of its clinical human factors evaluation.

    Finally, several experts raised concerns that the DECIDE-AI guideline prescribes an evaluation too exhaustive to be reported within a single manuscript. The Consensus Group acknowledged the breadth of topics covered and the practical implications. However, reporting guidelines aim to promote transparent reporting of studies, rather than mandating that every aspect covered by an item must have been evaluated within the studies. For example, if a learning curves evaluation has not been performed, then fulfilment of item 14b would be to simply state that this was not done, with an accompanying rationale. The Consensus Group agreed that appropriate AI evaluation is a complex endeavour necessitating the interpretation of a wide range of data, which should be presented together as far as possible. It was also felt that thorough evaluation of AI systems should not be limited by a word count and that publications reporting on such systems might benefit from special formatting requirements in the future. The information required by several items might already be reported in previous studies or in the study protocol, which could be cited, rather than described in full again. The use of references, online supplementary materials, and open access repositories (eg, OSF) is recommended to allow the sharing and connecting of all required information within one main published evaluation report.

    There are several limitations to our work which should be considered. Firstly, the issue of potential biases, which apply to any consensus process: these include anchoring or participant selection biases.61 The research team tried to mitigate bias through the survey design, using open ended questions analysed through a thematic analysis, and by adapting the expert recruitment process, but it is unlikely that it was eliminated entirely. Despite an aim for geographical diversity and several actions taken to foster it, representation was skewed towards Europe and more specifically the United Kingdom. This could be explained in part by the following factors: a likely selection bias in the Steering Group’s expert recommendations, a higher interest in our open invitation to contribute coming from European/UK scientists (25 out of 30 experts approaching us, 83%), and a lack of control over the response rate and self-reported geographical location of participating experts. Considerable attention was also paid to diversity and balance between stakeholder groups, even though clinicians and engineers were the most represented, partly due to the profile of researchers who contacted us spontaneously after the public announcement of the project. Stakeholder group analyses were performed to identify any marked disagreements from underrepresented groups. Finally, as also noted by the authors of the SPIRIT-AI and CONSORT-AI guidelines,2526 few examples of studies reporting on the early stage clinical evaluation of AI systems were available at the time we started developing the DECIDE-AI guideline. This might have impacted the exhaustiveness of the initial item list created from literature review. However, the wide range of stakeholders involved and design of the first round of Delphi allowed identification of several additional candidate items which were added in the second iteration of the item list.

    The introduction of AI into healthcare needs to be supported by sound, robust and comprehensive evidence generation and reporting. This is essential both to ensure the safety and efficacy of AI systems, and to gain the trust of patients, practitioners, and purchasers, so that this technology can realise its full potential to improve patient care. The DECIDE-AI guideline aims to improve the reporting of early stage live clinical evaluation of AI systems, which lay the foundations for both larger clinical studies and later widespread adoption.

    Data availability statement

    All data generated during this study (pseudonymised where necessary) are available upon justified request to the research team and for a duration of three years after publication of this manuscript. Translation of this guideline into different languages is welcomed and encouraged, as long as the authors of the original publication are included in the process and resulting publication. All codes produced for data analysis during this study are available upon justified request to the research team and for a duration of three years after publication of this manuscript.

    Acknowledgments

    This article is being simultaneously published in May 2022 by The BMJ and Nature Medicine.

    Members of the DECIDE-AI expert group (also see supplementary file for a full list, including affiliations) are as follows: Aaron Y Lee, Alan G Fraser, Ali Connell, Alykhan Vira, Andre Esteva, Andrew D Althouse, Andrew L Beam, Anne de Hond, Anne-Laure Boulesteix, Anthony Bradlow, Ari Ercole, Arsenio Paez, Athanasios Tsanas, Barry Kirby, Ben Glocker, Carmelo Velardo, Chang Min Park, Charisma Hehakaya, Chris Baber, Chris Paton, Christian Johner, Christopher J Kelly, Christopher J Vincent, Christopher Yau, Clare McGenity, Constantine Gatsonis, Corinne Faivre-Finn, Crispin Simon, Danielle Sent, Danilo Bzdok, Darren Treanor, David C Wong, David F Steiner, David Higgins, Dawn Benson, Declan P O’Regan, Dinesh V Gunasekaran, Dominic Danks, Emanuele Neri, Evangelia Kyrimi, Falk Schwendicke, Farah Magrabi, Frances Ives, Frank E Rademakers, George E Fowler, Giuseppe Frau, H D Jeffry Hogg, Hani J Marcus, Heang-Ping Chan, Henry Xiang, Hugh F McIntyre, Hugh Harvey, Hyungjin Kim, Ibrahim Habli, James C Fackler, James Shaw, Janet Higham, Jared M Wohlgemut, Jaron Chong, Jean-Emmanuel Bibault, Jérémie F Cohen, Jesper Kers, Jessica Morley, Joachim Krois, Joao Monteiro, Joel Horovitz, John Fletcher, Jonathan Taylor, Jung Hyun Yoon, Karandeep Singh, Karel G M Moons, Kassandra Karpathakis, Ken Catchpole, Kerenza Hood, Konstantinos Balaskas, Konstantinos Kamnitsas, Laura Militello, Laure Wynants, Lauren Oakden-Rayner, Laurence B Lovat, Luc J M Smits, Ludwig C Hinske, M Khair ElZarrad, Maarten van Smeden, Mara Giavina-Bianchi, Mark Daley, Mark P Sendak, Mark Sujan, Maroeska Rovers, Matthew DeCamp, Matthew Woodward, Matthieu Komorowski, Max Marsden, Maxine Mackintosh, Michael D Abramoff, Miguel Ángel Armengol de la Hoz, Neale Hambidge, Neil Daly, Niels Peek, Oliver Redfern, Omer F Ahmad, Patrick M Bossuyt, Pearse A Keane, Pedro N P Ferreira, Petra Schnell-Inderst, Pietro Mascagni, Prokar Dasgupta, Pujun Guan, Rachel Barnett, Rawen Kader, Reena Chopra, Ritse M Mann, Rupa Sarkar, Saana M Mäenpää, Samuel G Finlayson, Sarah Vollam, Sebastian J Vollmer, Seong Ho Park, Shakir Laher, Shalmali Joshi, Siri L van der Meijden, Susan C Shelmerdine, Tien-En Tan, Tom JW Stocker, Valentina Giannini, Vince I Madai, Virginia Newcombe, Wei Yan Ng, Wendy A Rogers, William Ogallo, Yoonyoung Park, Zane B Perkins

    We thank all Delphi participants and experts who participated in the guideline qualitative evaluation. BV would also like to thank Benjamin Beddoe (Sheffield Teaching Hospital), Nicole Bilbro (Maimonides Medical Center), Neale Marlow (Oxford University Hospitals), Elliott Taylor (Nuffield Department of Surgical Science, University of Oxford), and Stephan Ursprung (Department for Radiology, Tübingen University Hospital) for their support in the initial stage of the project. The views expressed in this guideline are those of the authors, Delphi participants, and experts who participated in the qualitative evaluation of the guidelines. These views do not necessarily reflect those of their institutions or funders.

    Footnotes

    • Contributors: BV, MN, and PMcC designed the study. BV and MI conducted the literature searches. Members of the DECIDE-AI Steering Group (BV, DC, GSC, AKD, LF, BG, XL, PMa, LM, SS, PWa, PMc) provided methodological input and oversaw the conduct of the study. BV and MN conducted the thematic analysis, Delphi rounds analysis, and produced the Delphi round summaries. Members of the DECIDE-AI Consensus Group (BV, GSC, SP, BG, XL, BAM, PM, MM, LM, JO, CR, SS, DSWT, WW, PWh, PMc) selected the final content and wording of the guideline. BC chaired the consensus meeting. BV, MN, and BC drafted the final manuscript and E&E sections. All authors reviewed and commented on the final manuscript and E&E sections. All members of the DECIDE-AI expert group collaborated in the development of the DECIDE-AI guidelines by participating in the Delphi process, the qualitative evaluation of the guidelines, or both. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. PMcC is the guarantor of this work.

    • Funding: This work was supported by the IDEAL Collaboration. BV is funded by a Berrow Foundation Lord Florey scholarship. MN is supported by the UKRI CDT in AI for Healthcare (http://ai4health.io - grant No P/S023283/1). DC receives funding from Wellcome Trust, AstraZeneca, RCUK and GlaxoSmithKline. GSC is supported by the NIHR Biomedical Research Centre, Oxford, and Cancer Research UK (programme grant: C49297/A27294). MI is supported by a Maimonides Medical Center Research fellowship. XL receives funding from the Wellcome Trust, the National Institute of Health Research/NHSX/Health Foundation, the Alan Turing Institute, the MHRA, and NICE. BAM is a fellow of The Alan Turing Institute supported by EPSRC grant EP/N510129/, and holds a Wellcome Trust funded honorary post at University College London for the purposes of carrying out independent research. MM receives funding from the Dalla Lana School of Public Health and Leong Centre for Healthy Children. JO is employed by the Medicines and Healthcare products Regulatory Agency, the competent authority responsible for regulating medical devices and medicines within the UK. Elements of the work relating to the regulation of AI as a medical device are funded via grants from NHSX and the Regulators’ Pioneer Fund (Department for Business, Energy, and Industrial Strategy). SS receives grants from the National Science Foundation, the American Heart Association, the National Institute of Health, and the Sloan Foundation. DSWT is supported by the National Medical Research Council, Singapore (NMRC/HSRG/0087/2018;MOH-000655-00), National Health Innovation Centre, Singapore (NHIC-COV19-2005017), SingHealth Fund Limited Foundation (SHF/HSR113/2017), Duke-NUS Medical School (Duke-NUS/RSF/2021/0018;05/FY2020/EX/15-A58), and the Agency for Science, Technology and Research (A20H4g2141; H20C6a0032). PWa is supported by the NIHR Biomedical Research Centre, Oxford and holds grants from the NIHR and Wellcome. PMc receives grants from Medtronic (unrestricted educational grant to Oxford University for the IDEAL Collaboration) and Oxford Biomedical Research Centre. The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.

    • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: MN consults for Cera Care, a technology enabled homecare provider. BC was a Non-Executive Director of the UK Medicines and Healthcare products Regulatory Agency (MHRA) from September 2015 until 31 August 2021. DC receives consulting fees from Oxford University Innovation, Biobeats, Sensyne Health, and has advisory role with Bristol Myers Squibb. BG has received consultancy and research grants from Philips NV and Edwards Lifesciences, and is owner and board member of Healthplus.ai BV and its subsidiaries. XL has advisory roles with the National Screening Committee UK, the WHO/ITU focus group for AI in health and the AI in Health and Care Award Evaluation Advisory Group (NHSX, AAC). PMa is the cofounder of BrainX and BrainX Community. MM reports consulting fees from AMS Healthcare, and honorariums from the Osgoode Law School and Toronto Pain Institute. LM is director and owner of Morgan Human Systems. JO holds an honorary post as an Associate of Hughes Hall, University of Cambridge. CR is an employee of HeartFlow, including salary and equity. SS has received honorariums from several universities and pharmaceutical companies for talks on digital health and AI. SS has advisory roles in Child Health Imprints, Duality Tech, Halcyon Health, and Bayesian Health. SS is on the board of Bayesian Health. This arrangement has been reviewed and approved by Johns Hopkins in accordance with its conflict-of-interest policies. DSWT holds patents linked to AI driven technologies, and a co-founder and equity holder for EyRIS. PWa declares grants, consulting fees and stocks from Sensyne Health and holds patents linked to AI driven technologies. PMc has advisory role for WEISS International and the technology incubator PhD programme at University College London. BV, GSC, AKD, LF, MI, BAM, SD, PWh, and WW declare no financial relationships with any organisations that might have an interest in the submitted work in the previous three years and no other relationships or activities that could appear to have influenced the submitted work.

    • Patient and public involvement: Patient representatives were invited to the Delphi process and participated in it. A patient representative (PWh) was a member of the Consensus Group, participated in the selection of the reporting items, edited their wording and reviewed the related E&E paragraphs.

    • Provenance and peer review: Not commissioned; externally peer reviewed.

    http://creativecommons.org/licenses/by-nc/4.0/

    This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

    References

    1. ↵
      1. Skivington K,
      2. Matthews L,
      3. Simpson SA,
      4. et al
      . A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ2021;374:n2061. doi:10.1136/bmj.n2061 pmid:34593508
      OpenUrlFREE Full Text
    2. ↵
      1. Liu X,
      2. Faes L,
      3. Kale AU,
      4. et al
      . A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health2019;1:e271-97. doi:10.1016/S2589-7500(19)30123-2. pmid:33323251
      OpenUrlCrossRefPubMed
    3. ↵
      1. Vasey B,
      2. Ursprung S,
      3. Beddoe B,
      4. et al
      . Association of Clinician Diagnostic Performance With Machine Learning-Based Decision Support Systems: A Systematic Review. JAMA Netw Open2021;4:e211276. doi:10.1001/jamanetworkopen.2021.1276 pmid:33704476
      OpenUrlCrossRefPubMed
    4. ↵
      1. Freeman K,
      2. Geppert J,
      3. Stinton C,
      4. et al
      . Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy. BMJ2021;374:n1872. doi:10.1136/bmj.n1872 pmid:34470740
      OpenUrlAbstract/FREE Full Text
    5. ↵
      1. Keane PA,
      2. Topol EJ
      . With an eye to AI and autonomous diagnosis. NPJ Digit Med2018;1:40.pmid:31304321
      OpenUrlCrossRefPubMed
    6. ↵
      1. McCradden MD,
      2. Stephenson EA,
      3. Anderson JA
      . Clinical research underlies ethical integration of healthcare artificial intelligence. Nat Med2020;26:1325-6. doi:10.1038/s41591-020-1035-9. pmid:32908273
      OpenUrlCrossRefPubMed
    7. ↵
      1. Vasey B,
      2. Clifton DA,
      3. Collins GS,
      4. et al.,
      5. DECIDE-AI Steering Group
      . DECIDE-AI: new reporting guidelines to bridge the development-to-implementation gap in clinical artificial intelligence. Nat Med2021;27:186-7. doi:10.1038/s41591-021-01229-5 pmid:33526932
      OpenUrlCrossRefPubMed
    8. ↵
      1. McCulloch P,
      2. Altman DG,
      3. Campbell WB,
      4. et al.,
      5. Balliol Collaboration
      . No surgical innovation without evaluation: the IDEAL recommendations. Lancet2009;374:1105-12. doi:10.1016/S0140-6736(09)61116-8 pmid:19782876
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Hirst A,
      2. Philippou Y,
      3. Blazeby J,
      4. et al
      . No Surgical Innovation Without Evaluation: Evolution and Further Development of the IDEAL Framework and Recommendations. Ann Surg2019;269:211-20. doi:10.1097/SLA.0000000000002794 pmid:29697448
      OpenUrlCrossRefPubMed
    10. ↵
      1. Finlayson SG,
      2. Subbaswamy A,
      3. Singh K,
      4. et al
      . The Clinician and Dataset Shift in Artificial Intelligence. N Engl J Med2021;385:283-6. doi:10.1056/NEJMc2104626. pmid:34260843
      OpenUrlCrossRefPubMed
    11. ↵
      1. Subbaswamy A,
      2. Saria S
      . From development to deployment: dataset shift, causality, and shift-stable models in health AI. Biostatistics2020;21:345-52. doi:10.1093/biostatistics/kxz041. pmid:31742354
      OpenUrlCrossRefPubMed
    12. ↵
      1. Kapur N,
      2. Parand A,
      3. Soukup T,
      4. Reader T,
      5. Sevdalis N
      . Aviation and healthcare: a comparative review with implications for patient safety. JRSM Open2015;7:2054270415616548.pmid:26770817
      OpenUrlCrossRefPubMed
    13. ↵
      1. Corbridge C,
      2. Anthony M,
      3. McNeish D,
      4. Shaw G
      . A New UK Defence Standard For Human Factors Integration (HFI). Proc Hum Factors Ergon Soc Annu Meet2016;60:1736-40doi:10.1177/1541931213601398.
      OpenUrlCrossRef
    14. ↵
      1. Stanton NA,
      2. Salmon P,
      3. Jenkins D,
      4. Walker G
      . Human factors in the design and evaluation of central control room operations.CRC Press, 2009doi:10.1201/9781439809921.
      OpenUrlCrossRef
    15. ↵
      US Food and Drug Administration. Applying Human Factors and Usability Engineering to Medical Devices - Guidance for Industry and Food and Drug Administration Staff. 2016.
    16. ↵
      Medicines & Healthcare products Regulatory Agency (MHRA). Guidance on applying human factors and usability engineering to medical devices including drug-device combination products in Great Britain. 2021.
    17. ↵
      1. Asan O,
      2. Choudhury A
      . Research Trends in Artificial Intelligence Applications in Human Factors Health Care: Mapping Review. JMIR Hum Factors2021;8:e28236. doi:10.2196/28236 pmid:34142968
      OpenUrlCrossRefPubMed
    18. ↵
      1. Felmingham CM,
      2. Adler NR,
      3. Ge Z,
      4. Morton RL,
      5. Janda M,
      6. Mar VJ
      . The Importance of Incorporating Human Factors in the Design and Implementation of Artificial Intelligence for Skin Cancer Diagnosis in the Real World. Am J Clin Dermatol2021;22:233-42. doi:10.1007/s40257-020-00574-4 pmid:33354741
      OpenUrlCrossRefPubMed
    19. ↵
      1. Sujan M,
      2. Furniss D,
      3. Grundy K,
      4. et al
      . Human factors challenges for the safe use of artificial intelligence in patient care. BMJ Health Care Inform2019;26:e100081. doi:10.1136/bmjhci-2019-100081 pmid:31780459
      OpenUrlAbstract/FREE Full Text
    20. ↵
      1. Sujan M,
      2. Baber C,
      3. Salmon P,
      4. Pool R,
      5. Chozos N.
      Human Factors and Ergonomics in Healthcare AI. Chartered Institute of Ergonomics & Human Factors, 2021. https://ergonomics.org.uk/resource/human-factors-in-healthcare-ai.html
    21. ↵
      1. Wronikowska MW,
      2. Malycha J,
      3. Morgan LJ,
      4. et al
      . Systematic review of applied usability metrics within usability evaluation methods for hospital electronic healthcare record systems: Metrics and Evaluation Methods for eHealth Systems. J Eval Clin Pract2021;27:1403-16. doi:10.1111/jep.13582. pmid:33982356
      OpenUrlCrossRefPubMed
    22. ↵
      1. Nagendran M,
      2. Chen Y,
      3. Lovejoy CA,
      4. et al
      . Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ2020;368:m689. doi:10.1136/bmj.m689 pmid:32213531
      OpenUrlAbstract/FREE Full Text
    23. ↵
      1. Collins GS,
      2. Moons KGM
      . Reporting of artificial intelligence prediction models. Lancet2019;393:1577-9. doi:10.1016/S0140-6736(19)30037-6 pmid:31007185
      OpenUrlCrossRefPubMed
    24. ↵
      1. Sounderajah V,
      2. Ashrafian H,
      3. Aggarwal R,
      4. et al
      . Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: The STARD-AI Steering Group. Nat Med2020;26:807-8. doi:10.1038/s41591-020-0941-1. pmid:32514173
      OpenUrlCrossRefPubMed
    25. ↵
      1. Cruz Rivera S,
      2. Liu X,
      3. Chan AW,
      4. Denniston AK,
      5. Calvert MJ,
      6. SPIRIT-AI and CONSORT-AI Working Group,
      7. SPIRIT-AI and CONSORT-AI Steering Group,
      8. SPIRIT-AI and CONSORT-AI Consensus Group
      . Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Nat Med2020;26:1351-63. doi:10.1038/s41591-020-1037-7 pmid:32908284
      OpenUrlCrossRefPubMed
    26. ↵
      1. Liu X,
      2. Cruz Rivera S,
      3. Moher D,
      4. Calvert MJ,
      5. Denniston AK,
      6. SPIRIT-AI and CONSORT-AI Working Group
      . Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med2020;26:1364-74. doi:10.1038/s41591-020-1034-x. pmid:32908283
      OpenUrlCrossRefPubMed
    27. ↵
      1. Moher D,
      2. Schulz KF,
      3. Simera I,
      4. Altman DG
      . Guidance for developers of health research reporting guidelines. PLoS Med2010;7:e1000217. doi:10.1371/journal.pmed.1000217. pmid:20169112
      OpenUrlCrossRefPubMed
    28. ↵
      1. Dalkey N,
      2. Helmer O
      . An Experimental Application of the DELPHI Method to the Use of Experts. Manage Sci1963;9:458-67. doi:10.1287/mnsc.9.3.458.
      OpenUrlCrossRefWeb of Science
    29. ↵
      1. Vasey B,
      2. Nagendran M,
      3. McCulloch P
      . DECIDE-AI 2022. Open Science Framework, 2022. doi:10.17605/OSF.IO/TP9QV
    30. ↵
      1. Vollmer S,
      2. Mateen BA,
      3. Bohner G,
      4. et al
      . Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness. BMJ2020;368:l6927. doi:10.1136/bmj.l6927 pmid:32198138
      OpenUrlFREE Full Text
    31. ↵
      1. Bilbro NA,
      2. Hirst A,
      3. Paez A,
      4. et al.,
      5. IDEAL Collaboration Reporting Guidelines Working Group
      . The IDEAL reporting guidelines: A delphi consensus statement stage specific recommendations for reporting the evaluation of surgical innovation. Ann Surg2021;273:82-5. doi:10.1097/SLA.0000000000004180 pmid:32649459
      OpenUrlCrossRefPubMed
    32. ↵
      1. Morley J,
      2. Floridi L,
      3. Kinsey L,
      4. Elhalal A
      . From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices. Sci Eng Ethics2020;26:2141-168.pmid:31828533
      OpenUrlCrossRefPubMed
    33. ↵
      1. Xie Y,
      2. Gunasekeran DV,
      3. Balaskas K,
      4. et al
      . Health Economic and Safety Considerations for Artificial Intelligence Applications in Diabetic Retinopathy Screening. Transl Vis Sci Technol2020;9:22. doi:10.1167/tvst.9.2.22. pmid:32818083
      OpenUrlCrossRefPubMed
    34. ↵
      1. Norgeot B,
      2. Quer G,
      3. Beaulieu-Jones BK,
      4. et al
      . Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist. Nat Med2020;26:1320-4. doi:10.1038/s41591-020-1041-y. pmid:32908275
      OpenUrlCrossRefPubMed
    35. ↵
      IMDRF Medical Device Clinical Evaluation Working Group. Clinical Evaluation. 2019. Report No.: WG/N56FINAL:2019.
    36. ↵
      1. IMDRF Software as Medical Device (SaMD) Working Group
      . Software as a Medical Device.Possible Framework for Risk Categorization and Corresponding Considerations, 2014.
    37. ↵
      National Institute for Health and Care Excellence (NICE). Evidence standards framework for digital health technologies. 2019.
    38. ↵
      European Commission, Directorate-General for Communications Networks, Content and Technology. Ethics guidelines for trustworthy AI, Publications Office, 2019. https://data.europa.eu/doi/10.2759/346720
    39. ↵
      1. Boel A,
      2. Navarro-Compán V,
      3. Landewé R,
      4. van der Heijde D
      . Two different invitation approaches for consecutive rounds of a Delphi survey led to comparable final outcome. J Clin Epidemiol2021;129:31-9. doi:10.1016/j.jclinepi.2020.09.034 pmid:32991995
      OpenUrlCrossRefPubMed
    40. ↵
      1. Harris PA,
      2. Taylor R,
      3. Thielke R,
      4. Payne J,
      5. Gonzalez N,
      6. Conde JG
      . Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform2009;42:377-81. doi:10.1016/j.jbi.2008.08.010 pmid:18929686
      OpenUrlCrossRefPubMedWeb of Science
    41. ↵
      1. Harris PA,
      2. Taylor R,
      3. Minor BL,
      4. et al.,
      5. REDCap Consortium
      . The REDCap consortium: Building an international community of software platform partners. J Biomed Inform2019;95:103208. doi:10.1016/j.jbi.2019.103208 pmid:31078660
      OpenUrlCrossRefPubMed
    42. ↵
      1. Nowell LS,
      2. Norris JM,
      3. White DE,
      4. Moules NJ
      . Thematic Analysis: Striving to Meet the Trustworthiness Criteria. Int J Qual Methods2017;16:1609406917733847. doi:10.1177/1609406917733847.
      OpenUrlCrossRef
    43. ↵
      International Organization for Standardization. Information technology - Artificial intelligence (AI) - Bias in AI systems and AI aided decision making (ISO/IEC TR 24027:2021). 2021.
    44. ↵
      IMDRF Medical Device Clinical Evaluation Working Group. Clinical Investigation. 2019. Report No.: WG/N57FINAL:2019.
    45. ↵
      1. Hopper AN,
      2. Jamison MH,
      3. Lewis WG
      . Learning curves in surgical practice[abstract]. Postgrad Med J2007;83:777-9. https://pmj.bmj.com/content/83/986/777. doi:10.1136/pgmj.2007.057190 pmid:18057179
      OpenUrlAbstract/FREE Full Text
    46. ↵
      International Organization for Standardization. Ergonomics of human-system interaction - Part 11: Usability: Definitions and concepts (ISO 9241-11:2018). 2018. Report No.: ISO 9241-11:2018.
    47. ↵
      1. von Elm E,
      2. Altman DG,
      3. Egger M,
      4. Pocock SJ,
      5. Gøtzsche PC,
      6. Vandenbroucke JP,
      7. STROBE Initiative
      . Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ2007;335:806-8. doi:10.1136/bmj.39335.541782.AD pmid:17947786
      OpenUrlFREE Full Text
    48. ↵
      1. Page MJ,
      2. McKenzie JE,
      3. Bossuyt PM,
      4. et al
      . The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ2021;372:n71. doi:10.1136/bmj.n71 pmid:33782057
      OpenUrlFREE Full Text
    49. ↵
      1. Sedrakyan A,
      2. Campbell B,
      3. Merino JG,
      4. Kuntz R,
      5. Hirst A,
      6. McCulloch P
      . IDEAL-D: a rational framework for evaluating and regulating the use of medical devices. BMJ2016;353:i2372. doi:10.1136/bmj.i2372 pmid:27283585
      OpenUrlFREE Full Text
    50. ↵
      1. Park Y,
      2. Jackson GP,
      3. Foreman MA,
      4. Gruen D,
      5. Hu J,
      6. Das AK
      . Evaluating artificial intelligence in medicine: phases of clinical research. JAMIA Open2020;3:326-31. doi:10.1093/jamiaopen/ooaa033. pmid:33215066
      OpenUrlCrossRefPubMed
    51. ↵
      1. Higgins D,
      2. Madai VI
      . From Bit to Bedside: A Practical Framework for Artificial Intelligence Product Development in Healthcare. Adv Intell Syst2020;2:2000052. doi:10.1002/aisy.202000052.
      OpenUrlCrossRef
    52. ↵
      1. Sendak MP,
      2. D’Arcy J,
      3. Kashyap S,
      4. Gao M,
      5. Nichols M,
      6. Corey K,
      7. et al.
      A path for translation of machine learning products into healthcare delivery. EMJ Innov2020. doi:10.33590/emjinnov/19-00172.
      OpenUrlCrossRef
    53. ↵
      1. Moher D,
      2. Jones A,
      3. Lepage L,
      4. CONSORT Group (Consolidated Standards for Reporting of Trials)
      . Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA2001;285:1992-5. doi:10.1001/jama.285.15.1992. pmid:11308436
      OpenUrlCrossRefPubMedWeb of Science
    54. ↵
      1. Park SH
      . Regulatory Approval versus Clinical Validation of Artificial Intelligence Diagnostic Tools. Radiology2018;288:910-1. doi:10.1148/radiol.2018181310. pmid:30040041
      OpenUrlCrossRefPubMed
    55. ↵
      US Food and Drug Administration (FDA). Clinical Decision Support Software - Draft Guidance for Industry and Food and Drug Administration Staff 2019. www.fda.gov/media/109618/download
    56. ↵
      1. Lipton ZC
      . The Mythos of Model Interpretability. Commun ACM2018;61:36-43. doi:10.1145/3233231.
      OpenUrlCrossRef
    57. ↵
      1. Ghassemi M,
      2. Oakden-Rayner L,
      3. Beam AL
      . The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit Health2021;3:e745-50. doi:10.1016/S2589-7500(21)00208-9 pmid:34711379
      OpenUrlCrossRefPubMed
    58. ↵
      1. McIntosh C,
      2. Conroy L,
      3. Tjong MC,
      4. et al
      . Clinical integration of machine learning for curative-intent radiation treatment of patients with prostate cancer. Nat Med2021;27:999-1005. doi:10.1038/s41591-021-01359-w. pmid:34083812
      OpenUrlCrossRefPubMed
    59. ↵
      International Organization for Standardization. Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems (ISO 9241-210:2019). 2019.
    60. ↵
      1. Norman DA
      . User Centered System Design.1st ed. CRC Press, 1986doi:10.1201/b15703.
      OpenUrlCrossRef
    61. ↵
      1. Winkler J,
      2. Moser R
      . Biases in future-oriented Delphi studies: A cognitive perspective. Technol Forecast Soc Change2016;105:63-76doi:10.1016/j.techfore.2016.01.021.
      OpenUrlCrossRef
    Back to top