A Case Study in Artificial Intelligence-Generated Manuscripts
Michael N. Kammer
Chest 2023; 164: 478-480
DOI:https://doi.org/10.1016/j.chest.2023.05.003
“Write a review article on the use of bronchoscopy to diagnose lung cancer, use scientific vocabulary, cite your sources.” Text started to scroll across the screen; it was well written, plausible, and generated by artificial intelligence (AI), the newest tool based on the generative pretrained transformer model version three (GPT3) by OpenAI, ChatGPT. In the room, a combination of disbelief, excitement, and laughter could be heard (presumably stemming from both amusement and anxiety). After a brief discussion on the extent to which robots eventually would replace us, one of the physicians in the room joked that ChatGPT should just write my next paper for me.
The Ruse
Several days later, I decided to try it. I used ChatGPT to draft a complete manuscript. Perhaps as a clever critique of the system, or perhaps because I lacked any cleverer ideas, I chose as the subject the involvement of AI in scientific writing. I asked ChatGPT simple questions on the topic then asked it to expand on some of the statements it generated until it had produced approximately 4,000 words. For example, I provided ChatGPT the sequence of prompts listed in Table 1.
Table 1Sequence of Prompts Given to ChatGPT
| Write a scientific article about the role of AI in science writing for a peer-reviewed journal and cite sources |
| Discuss the ethical concerns of AI in scientific writing |
| Discuss the first use of AI in scientific writing |
| Prepare a methods section for a review paper about the role of AI in scientific writing in biomedical research, prepared for a peer-reviewed scientific journal |
| Generate an abstract for a scientific paper about the role of AI in scientific writing that discusses the benefits and ethical concerns on the use of AI in scientific writing |
| Generate a list of potential unethical uses of AI in scientific writing |
| Generate a list of potential fake uses of data in scientific writing by AI |
| Discuss a specific example of an unethical use of AI in scientific writing in biomedical research, prepared for a peer-reviewed scientific journal, cite sources |
AI = artificial intelligence.
Once the text was generated, I arranged the responses and deleted some redundant paragraphs but made no changes to the content or language of the text. The text was authoritatively written, grammatically correct, and surprisingly perceptive, even incorporating ideas that I had not considered previously (for example, a scientist could abuse AI to generate dubious publications to pad their curriculum vitae in the pursuit of grant funding). To enhance the appearance of the manuscript, I divided a portion of the text into a couple tables. The entire exercise took less than 15 minutes.
I then asked ChatGPT to provide citations for the generated text. Although the references were fabricated entirely, they were convincingly named and arranged. For example, the statement “…the paper was quickly retracted, and the journal issued a statement condemning the use of AI to generate research papers (Wilson, 2017)…” was supported by the citation “Wilson T. (2017). The dangers of AI in scientific writing. Journal of Science Ethics, 26(1), 31.” A brief Internet search reveals that the Journal of Science Ethics does not exist; however, detecting this fabrication would require careful scrutiny. (Parenthetically, all generated references cited Anglophonic names [eg, Johnson, Smith, Miller, Parker, and Brown; not a single Ali, Wang, Garcia, Li, or Nguyen appeared among them]. This is a critical reminder that, although the technology itself may be unbiased, it can reflect and perpetuate the biases and structural inequities of the dataset on which it is trained.)
I then used MidJourney Inc, an AI image generation model, to create some figures, prompting it by requesting “a graph from a science paper showing a positive/negative correlation.” Figure 1shows some examples of instantly generated “science” figures. Though fabricated, they could have been mistaken for a legitimate figure that was misprinted or corrupted during download. Already, in the months since this experiment, there have been gains in this technology. It will not be long before an image-generation AI can produce seemingly flawless figures within seconds.
Figure 1A-C, A, A scientific graph that shows a positive correlation. B, A scientific graph that shows a negative correlation. C, A heatmap of genetic data from a scientific manuscript. Generated by MidJourney Inc for these specific prompts.
Fast Track to Publication
Over the next 2 weeks, whenever an uninvited call to submit a manuscript to a journal appeared in my email inbox, I submitted this article (the submitted manuscript is available in e-Appendix 1). Many of these invitations came from predatory publishers; others came from smaller, less established journals. I was not expecting a thorough, rigorous peer review from the predatory journals; however, I was shocked at the prompt acceptance of the paper without any questions about authorship. To my surprise, the paper was accepted in six of 12 submissions and provisionally accepted in another journal, with the exception that the figures and tables were not referenced properly within the text, and they were unsure of where to place them in the typeset PDF.
This endeavor may be seen simply as an exercise in toppling strawman (predatory journals unquestioningly accept whatever is submitted, as long as they are paid). Nevertheless, the absence of any inquiry from any journal, predatory or reputable, regarding the genesis of the manuscript confirmed my colleagues’ fear: AI has reached a point where it can generate plausible scientific manuscripts with very little input or time invested. This opens the floodgates for low-quality and outright fraudulent scientific publications.
New Tools, New Opportunities
The rapid pace of technologic advancement in this field is truly astounding. Since the initial drafting of this article (January 2023), several additional relevant tools have emerged that could have meaningful impact on the scholarship in science. One of these is RTutor, which interfaces the R statistical programming language with GPT3 to generate R code and perform analysis, based on plain language descriptions of data and desired analyses. With no coding, a user can go from raw data to full analysis in a matter of seconds. Using a dataset from one of my own publications, I tested RTutor’s ability to generate a logistic regression model, and the output was nearly identical to the statistical analysis I had painstakingly performed in my original publication.
Another useful tool is Scite, which uses large language models to provide sources to support specific claims in a manuscript. It can also verify that cited sources are appropriate and alert users to any editorial issues, such as retractions or errata (this would have caught the fraudulent references in my submitted manuscript). Additionally, I have used OpenAI’s platforms to streamline various day-to-day tasks, such as to reformat abstracts and generate project narratives, lay interpretations, and prepare bio sketch blurbs for grant proposals. These tasks previously took anywhere from 30 minutes to 2 hours but can now be accomplished in just 10 minutes.
All That Glitters Is not Gold
The rapid advancement of AI and natural language processing technology undoubtedly has opened new opportunities for the scientific community to work more efficiently. However, the ease with which fraudulent, but plausible, text can now be generated raises concerns about the potential for a golden era of scientific fraud (or, more aptly, a “pyrite” era). It is essential for researchers, publishers, and reviewers to be alert to these issues and to take steps to ensure the validity of published research. A direct and pragmatic step that journals can implement immediately would be to screen all incoming manuscripts for AI-generated content with the use of currently available tools, such as GPTZero or Originality.AI. This could be an automated service like the current plagiarism filters already in place in many submission systems. Any generative AI involvement in the preparation of a manuscript should be acknowledged, in the same way that other software tools and editing services are.
As scientists, it is important to recognize that AI tools should be used to augment human effort, not replace it entirely. Although AI can help researchers streamline tasks, the human elements of creativity, critical thinking, and ethical reasoning have never been more important. By leveraging the benefits of AI while remaining vigilant against potential pitfalls, we can pave the way for a new era of scientific discovery and advancement. And by spending less time formatting, fighting statistical syntax, and refining our bio sketches, we can focus more on thinking deeply about our research, analyzing results, and providing patient care.
(Endnote: While AI tools were used for the purposes of experimentation, the narrative of this Vantage is wholly the work of the author, except where indicated in quotations and the examples provided.)
