Process-based Self-Rewarding Language Models
Shimao Zhang, Xiao Liu, Xin Zhang, Junxiao Liu, Zheheng Luo, Shujian Huang, Yeyun Gong
2025-03-06
Summary
This paper talks about a new way to make AI language models better at math problems by teaching them to check and improve their own work step-by-step
What's the problem?
Current AI language models are trained using human feedback, which limits how good they can get. Existing methods for AI to improve itself don't work well for math problems and can even make the AI worse at them
What's the solution?
The researchers created a new method called Process-based Self-Rewarding. This method makes the AI think through math problems step-by-step, check each step, and learn from its mistakes. The AI acts like its own teacher, grading its work and figuring out how to do better next time
Why it matters?
This matters because it could help AI become much better at solving complex math problems, potentially even better than humans. This could lead to breakthroughs in fields that rely on advanced math, like physics or engineering, and help create smarter AI systems that can reason more like humans do
Abstract
Large Language Models have demonstrated outstanding performance across various downstream tasks and have been widely applied in multiple scenarios. Human-annotated preference data is used for training to further improve LLMs' performance, which is constrained by the upper limit of human performance. Therefore, Self-Rewarding method has been proposed, where LLMs generate training data by rewarding their own outputs. However, the existing self-rewarding paradigm is not effective in mathematical reasoning scenarios and may even lead to a decline in performance. In this work, we propose the Process-based Self-Rewarding pipeline for language models, which introduces long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference optimization within the self-rewarding paradigm. Our new paradigm successfully enhances the performance of LLMs on multiple mathematical reasoning benchmarks through iterative Process-based Self-Rewarding, demonstrating the immense potential of self-rewarding to achieve LLM reasoning that may surpass human capabilities.