Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depth
Yang Wang, Chenghao Xiao, Chia-Yi Hsiao, Zi Yan Chang, Chi-Li Chen, Tyler Loakman, Chenghua Lin
2025-09-05
Summary
This paper introduces a concept called 'Drivelology,' which is essentially language that *sounds* like nonsense but actually has hidden layers of meaning, emotion, or a sneaky way of making a point. The researchers found that even though computer programs (specifically large language models or LLMs) are really good at understanding and using language in many ways, they consistently fail to understand this more complex, subtle type of communication.
What's the problem?
Current AI language models are really good at processing what words *literally* mean and how they're put together, but they struggle with understanding the *intent* behind language, especially when that intent isn't obvious. Drivelology relies on understanding things like sarcasm, hidden messages, or emotional context, and the researchers noticed LLMs just didn't 'get it.' There wasn't a good way to test this specifically, because identifying Drivelology is tricky even for humans.
What's the solution?
To address this, the researchers created a dataset of over 1,200 examples of Drivelology in multiple languages – English, Mandarin, Spanish, French, Japanese, and Korean. Getting these examples right was hard; they had experts carefully review each one to make sure it truly fit the definition. Then, they tested a bunch of different LLMs on this dataset, asking them to identify Drivelology, explain it, and figure out what it was trying to say. The results showed the LLMs consistently missed the mark, often thinking it was just random nonsense or giving illogical explanations.
Why it matters?
This research shows that while AI is getting better at *using* language, it still doesn't truly *understand* it in the same way humans do. Just because a computer can string words together convincingly doesn't mean it grasps the deeper meaning, emotional weight, or rhetorical tricks that humans use all the time. This highlights a gap in AI's ability to think and reason like people, and the researchers are sharing their dataset so others can work on improving AI's understanding of these more nuanced aspects of language.
Abstract
We introduce Drivelology, a unique linguistic phenomenon characterised as "nonsense with depth", utterances that are syntactically coherent yet pragmatically paradoxical, emotionally loaded, or rhetorically subversive. While such expressions may resemble surface-level nonsense, they encode implicit meaning requiring contextual inference, moral reasoning, or emotional interpretation. We find that current large language models (LLMs), despite excelling at many natural language processing (NLP) tasks, consistently fail to grasp the layered semantics of Drivelological text. To investigate this, we construct a small but diverse benchmark dataset of over 1,200 meticulously curated examples, with select instances in English, Mandarin, Spanish, French, Japanese, and Korean. Annotation was especially challenging: each of the examples required careful expert review to verify that it truly reflected Drivelological characteristics. The process involved multiple rounds of discussion and adjudication to address disagreements, highlighting the subtle and subjective nature of the Drivelology. We evaluate a range of LLMs on classification, generation, and reasoning tasks. Our results reveal clear limitations of LLMs: models often confuse Drivelology with shallow nonsense, produce incoherent justifications, or miss the implied rhetorical function altogether. These findings highlight a deeper representational gap in LLMs' pragmatic understanding and challenge the assumption that statistical fluency implies cognitive comprehension. We release our dataset and code to facilitate further research in modelling linguistic depth beyond surface-level coherence.