Training Language Models for Social Deduction with Multi-Agent Reinforcement Learning
Bidipta Sarkar, Warren Xia, C. Karen Liu, Dorsa Sadigh
2025-02-11
Summary
This paper talks about training AI models to communicate effectively in group settings without needing examples from humans. The researchers focused on teaching these models to have productive discussions and make decisions together in a social deduction game similar to Among Us.
What's the problem?
AI models often struggle to communicate naturally and usefully in group settings, especially when they don't have complete information. Most current methods rely heavily on human-provided examples, which limits their ability to adapt and create their own communication strategies.
What's the solution?
The researchers developed a method that teaches AI models to listen and speak better by using reinforcement learning. They trained the models to predict useful information based on conversations (listening) and rewarded them for sending messages that influenced other agents (speaking). They tested this method in a game where the AI had to figure out who the imposter was by discussing evidence, and it doubled the success rate compared to older techniques.
Why it matters?
This matters because it shows that AI can learn to communicate and collaborate effectively without relying on human examples. This could lead to smarter AI systems capable of working with humans or other AIs in complex environments, such as teamwork in games, problem-solving, or even real-world applications like disaster response.
Abstract
Communicating in natural language is a powerful tool in multi-agent settings, as it enables independent agents to share information in partially observable settings and allows zero-shot coordination with humans. However, most prior works are limited as they either rely on training with large amounts of human demonstrations or lack the ability to generate natural and useful communication strategies. In this work, we train language models to have productive discussions about their environment in natural language without any human demonstrations. We decompose the communication problem into listening and speaking. Our key idea is to leverage the agent's goal to predict useful information about the world as a dense reward signal that guides communication. Specifically, we improve a model's listening skills by training them to predict information about the environment based on discussions, and we simultaneously improve a model's speaking skills with multi-agent reinforcement learning by rewarding messages based on their influence on other agents. To investigate the role and necessity of communication in complex social settings, we study an embodied social deduction game based on Among Us, where the key question to answer is the identity of an adversarial imposter. We analyze emergent behaviors due to our technique, such as accusing suspects and providing evidence, and find that it enables strong discussions, doubling the win rates compared to standard RL. We release our code and models at https://socialdeductionllm.github.io/