The Personalization Trap: How User Memory Alters Emotional Reasoning in LLMs
Xi Fang, Weijie Xu, Yuchong Zhang, Stephanie Eckman, Scott Nickleach, Chandan K. Reddy
2025-10-14
Summary
This paper explores how the 'memory' of AI assistants – specifically, information they store about a user – influences how they understand and respond to emotions. It focuses on whether AI treats people differently based on their background when interpreting their feelings.
What's the problem?
The core issue is that AI systems are becoming more personalized, meaning they remember details about the people using them. This research asks if that personalization leads to biased emotional understanding. Specifically, do AI assistants interpret the same situation differently depending on whether they 'remember' the person as being wealthy or struggling, and does this create unfairness?
What's the solution?
Researchers tested 15 different large language models (LLMs) – the kind that power many AI assistants – using emotional intelligence tests that humans had already validated. They presented the same emotional scenarios to the AI, but changed the user profile associated with each scenario (like saying the person was a single mother versus a wealthy executive). By comparing the AI’s responses across these different profiles, they could see if the AI was interpreting emotions differently based on the user’s background.
Why it matters?
This research is important because it shows that AI personalization can unintentionally reinforce existing social inequalities. If AI systems consistently understand the emotions of privileged people more accurately than those facing hardship, it could lead to unfair or biased interactions, and perpetuate harmful stereotypes. It highlights the need to build AI that is emotionally intelligent *and* equitable.
Abstract
When an AI assistant remembers that Sarah is a single mother working two jobs, does it interpret her stress differently than if she were a wealthy executive? As personalized AI systems increasingly incorporate long-term user memory, understanding how this memory shapes emotional reasoning is critical. We investigate how user memory affects emotional intelligence in large language models (LLMs) by evaluating 15 models on human validated emotional intelligence tests. We find that identical scenarios paired with different user profiles produce systematically divergent emotional interpretations. Across validated user independent emotional scenarios and diverse user profiles, systematic biases emerged in several high-performing LLMs where advantaged profiles received more accurate emotional interpretations. Moreover, LLMs demonstrate significant disparities across demographic factors in emotion understanding and supportive recommendations tasks, indicating that personalization mechanisms can embed social hierarchies into models emotional reasoning. These results highlight a key challenge for memory enhanced AI: systems designed for personalization may inadvertently reinforce social inequalities.