How Much Research Is Being Written by Large Language Models?

🌈 Abstract

The article discusses the increasing use of large language models (LLMs) in academic writing, including in scientific papers and peer reviews. It presents findings from two studies conducted by a team led by James Zou, which analyzed the prevalence of AI-generated content in computer science papers and peer review text.

🙋 Q&A

[01] Determining AI-generated content

1. Questions related to the content of the section?

• The researchers identified specific words that are more likely to be used by LLMs than by humans, such as "commendable", "innovative", "meticulous", "pivotal", "intricate", "realm", and "showcasing". The sharp spike in the frequency of these words in reviews, coinciding with the release of ChatGPT, provided strong evidence that the reviews contained AI-generated content.
• The researchers conducted an experiment where they used LLMs to write reviews of papers and compared them to human-written reviews, which allowed them to quantify the words more likely to be used by LLMs versus humans.

[02] Ethics and policies around LLM use in academia

1. Questions related to the content of the section?

• The use of LLMs in academic writing is an important and timely topic, as journal policies are changing quickly. Some journals, like Science, initially prohibited the use of LLMs but later changed their policies to allow it, as long as the authors explicitly note where the LLM was used.
• Journals are struggling to define the right way to handle the use of LLMs in academic writing, as the policies continue to evolve.

[03] Differences in LLM usage across disciplines

1. Questions related to the content of the section?

• The researchers found that computer science and AI-related disciplines had a higher rate of LLM usage in papers (up to 17.5%) compared to math and Nature family papers (around 6.3%).
• This discrepancy is likely due to the explosion in the number of papers submitted to AI and computer science conferences, which has created a burden on reviewers and authors, potentially leading some authors to use LLMs to keep up with the increased competition.

[04] Scope of the study and data sources

1. Questions related to the content of the section?

• The researchers analyzed close to a million papers from arXiv, bioRxiv, and Nature, focusing primarily on computer science, engineering, and biomedical areas, as well as interdisciplinary areas like the Nature family journals.
• The researchers were able to obtain data from these sources, but noted that it was more difficult to access data from humanities journals, which limited the scope of their analysis.

[05] Surprising findings

1. Questions related to the content of the section?

• The researchers were surprised by the magnitude and speed of the increase in LLM usage, with nearly a fifth of papers and peer review text using some form of LLM modification.
• They also found that peer reviews submitted closer to the deadline and those less likely to engage with author rebuttal were more likely to use LLMs, suggesting that some reviewers may be offloading work to AI due to lack of engagement.

[06] Implications and recommendations

1. Questions related to the content of the section?

• LLMs are transforming how research is conducted, and there needs to be more transparency around their use in academic writing.
• Researchers should explicitly state how LLMs are used and if they are used substantially, as the human researchers should be accountable for the content in the papers.
• There are constructive ways to use LLMs in the research process, such as in earlier drafting stages, but it is important to maintain human oversight and rigorous evaluation.