Ghost in the machine or monkey with a typewriter—generating titles for Christmas research articles in The BMJ using artificial intelligence: observational study
BMJ 2021; 375 doi: https://doi.org/10.1136/bmj-2021-067732 (Published 15 December 2021) Cite this as: BMJ 2021;375:e067732
- Robin Marlow
, consultant in paediatric emergency medicine, honorary senior lecturer1 2, - Dora Wood, consultant in paediatric intensive care1
- 1Bristol Royal Hospital for Children, Bristol, BS2 8BJ, UK
- 2Centre for Academic Child Health, University of Bristol, Bristol, UK
- Correspondence to: R Marlow robin.marlow{at}bristol.ac.uk (or @robindmarlow on Twitter)
- Accepted 21 October 2021
Abstract
Objective To determine whether artificial intelligence (AI) can generate plausible and engaging titles for potential Christmas research articles in The BMJ.
Design Observational study.
Setting Europe, Australia, and Africa.
Participants 1 AI technology (Generative Pre-trained Transformer 3, GPT-3) and 25 humans.
Main outcome measures Plausibility, attractiveness, enjoyability, and educational value of titles for potential Christmas research articles in The BMJ generated by GPT-3 compared with historical controls.
Results AI generated titles were rated at least as enjoyable (159/250 responses (64%) v 346/500 responses (69%); odds ratio 0.9, 95% confidence interval 0.7 to 1.2) and attractive (176/250 (70%) v 342/500 (68%); 1.1, 0.8 to 1.4) as real control titles, although the real titles were rated as more plausible (182/250 (73%) v 238/500 (48%); 3.1, 2.3 to 4.1). The AI generated titles overall were rated as having less scientific or educational merit than the real controls (146/250 (58%) v 193/500 (39%); 2.0, 1.5 to 2.6); this difference, however, became non-significant when humans curated the AI output (146/250 (58%) v 123/250 (49%); 1.3, 1.0 to 1.8). Of the AI generated titles, the most plausible was “The association between belief in conspiracy theories and the willingness to receive vaccinations,” and the highest rated was “The effects of free gourmet coffee on emergency department waiting times: an observational study.”
Conclusions AI can generate plausible, entertaining, and scientifically interesting titles for potential Christmas research articles in The BMJ; as in other areas of medicine, performance was enhanced by human intervention.
Footnotes
Contributors: RM and DW designed the study and drafted the paper. They are both guarantors. RM collected and analysed the data. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding: None.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.docx and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
The corresponding author (RM) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; no important aspects of the study have been omitted; that any discrepancies from the study as planned have been explained.
Dissemination to participants and related patient and public communities: Results were sent to the participants after taking part. We will share our results with the wider community through social media channels, educational meetings, and press release.
Provenance and peer review: Not commissioned; externally peer reviewed.
Data availability statement
Dataset and full reproducible code are available at https://doi.org/10.5281/zenodo.5681251.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.