Research Christmas 2021: Time Warp

Ghost in the machine or monkey with a typewriter—generating titles for Christmas research articles in The BMJ using artificial intelligence: observational study

Robin Marlow, Dora Wood

BMJ 2021; 375 doi: https://doi.org/10.1136/bmj-2021-067732 (Published 15 December 2021)Cite this as: BMJ 2021;375:e067732

Abstract

Objective To determine whether artificial intelligence (AI) can generate plausible and engaging titles for potential Christmas research articles in The BMJ.

Design Observational study.

Setting Europe, Australia, and Africa.

Participants 1 AI technology (Generative Pre-trained Transformer 3, GPT-3) and 25 humans.

Main outcome measures Plausibility, attractiveness, enjoyability, and educational value of titles for potential Christmas research articles in The BMJ generated by GPT-3 compared with historical controls.

Results AI generated titles were rated at least as enjoyable (159/250 responses (64%) v 346/500 responses (69%); odds ratio 0.9, 95% confidence interval 0.7 to 1.2) and attractive (176/250 (70%) v 342/500 (68%); 1.1, 0.8 to 1.4) as real control titles, although the real titles were rated as more plausible (182/250 (73%) v 238/500 (48%); 3.1, 2.3 to 4.1). The AI generated titles overall were rated as having less scientific or educational merit than the real controls (146/250 (58%) v 193/500 (39%); 2.0, 1.5 to 2.6); this difference, however, became non-significant when humans curated the AI output (146/250 (58%) v 123/250 (49%); 1.3, 1.0 to 1.8). Of the AI generated titles, the most plausible was “The association between belief in conspiracy theories and the willingness to receive vaccinations,” and the highest rated was “The effects of free gourmet coffee on emergency department waiting times: an observational study.”

Conclusions AI can generate plausible, entertaining, and scientifically interesting titles for potential Christmas research articles in The BMJ; as in other areas of medicine, performance was enhanced by human intervention.