Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI Extension

Dear Editor

The recent BMJ paper, “Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI Extension” by SC Rivera, X Liu, et al (BMJ 2020; 370:m3210, DOI 10.1136/bmj.m3210, 9 Sep 2020) proposes a new standard protocol for interventional trials using AI.

Curiously, it mentions in passing a critical problem that it overlooks managing in its proposed protocol. The paper says that incremental software changes, intentional or unintentional, could have serious consequences on safety. It goes on to say that it is vitally important that such changes are documented and identified by software version.

Unfortunately, this is far from adequate for software that affects patient safety. International standards such as IEC 61508 require rigour in the way safety hazards are identified, documented and controlled; they require documented and rigorous review and verification of the delivered software and evidence that the software has not changed since the most recent verification.

The medical culture for thinking about AI (including ML) seems to be many decades behind where the technology is. This harsh claim might be illustrated by considering the new protocols requirements for statistics: namely, that named statistical methods must be in the protocol. This of course is only to be expected. Yet knowing a statistical method is pointless if the software code that generates the data or runs the analysis is itself unknown and not disclosed in the work.

In the old days when it originally made sense to require the statistical methods to be disclosed, we held in mind an image of a diligent peer reviewer deciding the method was appropriate, and then checking the data and statistics reported were correct. Today, peer reviewers run code. Unless the code is disclosed, they cannot scrutinise what it does. Unless the code is disclosed, they have no idea what it actually does.

In the extreme, the code could be totally fraudulent. It could check it is supplied with the research’s data, then it churns out contrived statistics to suit the agenda of the report or paper.

Unless the data and code are disclosed, particularly in AI where both are complex and the training data has a major influence on the software behaviour, errors (whether unintentional or intentional) cannot be detected either in review or replication. Moreover, without a professional software specification and evidence of verification and validation, almost all software is far too long and complex to scrutinise as it stands. In short, all protocols should require included network access to a version control repository that includes all the relevant material.

We trust that in future developments of protocols, the increasingly central and critical role of software — particularly in managing its defects and scrutinising its quality during clinical and other scientific work — will take on a prominent role in protocols.

Professor Harold Thimbleby, See Change Fellow in Digital Health, Swansea University

Professor Martyn Thomas, Fellow, Gresham College

Address for correspondence: harold@thimbleby.net