People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text
Jenna Russell, Marzena Karpinska, Mohit Iyyer
2025-01-30
Summary
This paper talks about how well people can spot text written by AI, especially those who often use AI writing tools like ChatGPT. The researchers tested this by having people read and judge a bunch of articles, deciding if they were written by humans or AI.
What's the problem?
As AI gets better at writing, it's becoming harder to tell the difference between AI-generated text and human-written text. This could lead to problems like fake news or academic cheating. We need to know if humans can still spot AI writing, and if so, what techniques they use.
What's the solution?
The researchers hired people to read 300 non-fiction articles and guess whether each was written by a human or AI. They found that people who frequently use AI writing tools are really good at spotting AI-generated text, even without special training. In fact, when they had five of these 'expert' users vote on each article, they only got 1 out of 300 wrong. These human experts were even better than most computer programs designed to detect AI writing. The researchers also looked at how these experts explained their decisions, finding that they used both simple clues (like specific words) and more complex judgments about the writing style.
Why it matters?
This matters because as AI writing becomes more common, we need ways to tell it apart from human writing. Knowing that frequent AI users are good at this task could help in developing better detection methods or training others to spot AI text. It could also help in fields like education or journalism where it's important to know if something was written by a human or AI. By sharing their data and methods, the researchers are helping others continue this important work in understanding and detecting AI-generated text.
Abstract
In this paper, we study how well humans can detect text generated by commercial LLMs (GPT-4o, Claude, o1). We hire annotators to read 300 non-fiction English articles, label them as either human-written or AI-generated, and provide paragraph-length explanations for their decisions. Our experiments show that annotators who frequently use LLMs for writing tasks excel at detecting AI-generated text, even without any specialized training or feedback. In fact, the majority vote among five such "expert" annotators misclassifies only 1 of 300 articles, significantly outperforming most commercial and open-source detectors we evaluated even in the presence of evasion tactics like paraphrasing and humanization. Qualitative analysis of the experts' free-form explanations shows that while they rely heavily on specific lexical clues ('AI vocabulary'), they also pick up on more complex phenomena within the text (e.g., formality, originality, clarity) that are challenging to assess for automatic detectors. We release our annotated dataset and code to spur future research into both human and automated detection of AI-generated text.