Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values
Dian Yu, Yulai Zhao, Kishan Panaganti, Linfeng Song, Haitao Mi, Dong Yu
2025-10-24
Summary
This paper introduces a new technique called Reinforcement Learning with Explicit Human Values, or RLEV, which aims to make large language models (LLMs) better at understanding and responding to what humans actually *value*, not just what is factually correct.
What's the problem?
Current methods for training LLMs, like using rewards for correct answers, don't account for the fact that some tasks or questions are more important than others. Just getting the right answer isn't enough; the model needs to understand *why* an answer matters. Existing systems focus on objective correctness, but real-world situations often involve subjective value judgements.
What's the solution?
RLEV solves this by directly incorporating human-defined values into the reward system used to train the LLM. The researchers used data where answers were labeled not only for correctness but also for their value or importance. This allows the model to learn to prioritize responses based on these value signals, resulting in answers that are both accurate *and* aligned with human priorities. They found the model learned to be more concise when the prompt wasn't very important, and more detailed when it was.
Why it matters?
This research is important because it offers a practical way to align LLMs with human preferences. Instead of just aiming for factual accuracy, RLEV allows us to teach models to consider the broader context and importance of a task, making them more helpful and reliable in real-world applications. It also shows that even if the value signals aren't perfect, the system can still learn effectively.
Abstract
We propose Reinforcement Learning with Explicit Human Values (RLEV), a method that aligns Large Language Model (LLM) optimization directly with quantifiable human value signals. While Reinforcement Learning with Verifiable Rewards (RLVR) effectively trains models in objective domains using binary correctness rewards, it overlooks that not all tasks are equally significant. RLEV extends this framework by incorporating human-defined value signals directly into the reward function. Using exam-style data with explicit ground-truth value labels, RLEV consistently outperforms correctness-only baselines across multiple RL algorithms and model scales. Crucially, RLEV policies not only improve value-weighted accuracy but also learn a value-sensitive termination policy: concise for low-value prompts, thorough for high-value ones. We demonstrate this behavior stems from value-weighted gradient amplification on end-of-sequence tokens. Ablation studies confirm the gain is causally linked to value alignment. RLEV remains robust under noisy value signals, such as difficulty-based labels, demonstrating that optimizing for an explicit utility function offers a practical path to aligning LLMs with human priorities.