Epistemic Diversity and Knowledge Collapse in Large Language Models
Dustin Wright, Sarah Masud, Jared Moore, Srishti Yadav, Maria Antoniak, Chan Young Park, Isabelle Augenstein
2025-10-07
Summary
This paper investigates whether large language models (LLMs) are becoming too similar in the information they provide, potentially limiting the range of knowledge people can access. It looks at how diverse the claims made by different LLMs are across various topics and cultures.
What's the problem?
LLMs are designed to predict the most likely text, and this can lead them to generate responses that are very similar to each other, even when there are multiple valid perspectives or pieces of information. This 'homogenization' of information is a problem because if everyone relies on LLMs that give the same answers, it could result in a 'knowledge collapse' where less common but still important information gets lost or ignored. Previous studies haven't really looked at this issue over time or across different cultures in a comprehensive way.
What's the solution?
The researchers developed a new method to measure 'epistemic diversity' – basically, how many different real-world claims an LLM makes about a topic. They then used this method to test 27 different LLMs on 155 topics from 12 different countries, using 200 different prompts based on actual user questions. This allowed them to see how diverse the responses were and how they changed with newer models, different model sizes, and the use of a technique called 'retrieval-augmented generation' (RAG).
Why it matters?
The study found that while newer LLMs are slightly more diverse, almost all of them still provide less diverse information than a simple web search. Surprisingly, larger models actually tended to be *less* diverse. Using RAG helped, but the amount of improvement depended on the cultural context. Importantly, the information LLMs provided about different countries often reflected English-language perspectives more than local ones. This highlights a real risk that LLMs could reinforce biases and limit access to a full range of knowledge, especially for information about non-English speaking regions.
Abstract
Large language models (LLMs) tend to generate lexically, semantically, and stylistically homogenous texts. This poses a risk of knowledge collapse, where homogenous LLMs mediate a shrinking in the range of accessible information over time. Existing works on homogenization are limited by a focus on closed-ended multiple-choice setups or fuzzy semantic features, and do not look at trends across time and cultural contexts. To overcome this, we present a new methodology to measure epistemic diversity, i.e., variation in real-world claims in LLM outputs, which we use to perform a broad empirical study of LLM knowledge collapse. We test 27 LLMs, 155 topics covering 12 countries, and 200 prompt variations sourced from real user chats. For the topics in our study, we show that while newer models tend to generate more diverse claims, nearly all models are less epistemically diverse than a basic web search. We find that model size has a negative impact on epistemic diversity, while retrieval-augmented generation (RAG) has a positive impact, though the improvement from RAG varies by the cultural context. Finally, compared to a traditional knowledge source (Wikipedia), we find that country-specific claims reflect the English language more than the local one, highlighting a gap in epistemic representation