The computer may be assessing you now, but who decided its values?

Machine learning holds great promise for medicine,1 but to assess its reliability we need to know what goals the machines have been directed to fulfil and whose values they represent. As Tim O’Reilly (originator of the term “web 2.0”) said in a recent blog, “Understanding how to evaluate algorithms without knowing the exact rules they follow is a key discipline in today’s world.”2 He set out four principles for doing this. Firstly, the algorithm’s creators have made clear what outcome they are seeking, and it is possible for external observers to verify that outcome. Secondly, success is measurable. Thirdly, the goals of the algorithm’s creators are aligned with the goals of the algorithm’s consumers. Finally, does the algorithm lead its creators and its users to make better longer term decisions?

This is a good start. But what happens when different values conflict? A drug firm funding a machine learning system might want to increase sales, whereas a healthcare system might want to hold down costs, and patients might prioritise safety.

One way forward would be to develop a public, agreed template for judging algorithms developed by machine learning in healthcare. This template would be an extension of the four rules outlined above and would be developed in an open and transparent process. Stakeholders might include algorithm generators, the NHS, clinicians, academics, and patients, although patients and their views are likely to be ignored in this process. A reputable body representing patient interests should therefore lead and hold other stakeholders to account as a consensus template is developed.

We need open standards urgently. Otherwise machine learning will build black boxes that reflect the values of venture capitalists and are completely opaque to the rest of us.