Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineering
Xinyan Guan, Yanjiang Liu, Xinyu Lu, Boxi Cao, Ben He, Xianpei Han, Le Sun, Jie Lou, Bowen Yu, Yaojie Lu, Hongyu Lin
2024-11-19
Summary
This paper discusses a new approach called verifier engineering, which aims to improve the training and performance of foundation models by providing better supervision through automated verification.
What's the problem?
Foundation models, which are large AI models trained on vast amounts of data, often struggle with receiving effective guidance or supervision during their training. This lack of effective supervision makes it difficult to enhance their capabilities further, leading to a need for new methods that can provide better feedback and support for these models.
What's the solution?
The authors propose a new framework called verifier engineering, which focuses on three main stages: search, verify, and feedback. This approach uses automated verifiers to check the model's outputs and provide meaningful feedback. By systematically categorizing the verification process, they aim to improve how foundation models learn and adapt. The paper reviews current research developments in each of these stages and highlights how this method can help advance AI towards achieving Artificial General Intelligence (AGI).
Why it matters?
This research is significant because it addresses a critical gap in the training of foundation models. By developing a structured way to provide better supervision through verifier engineering, the authors hope to enhance the performance of these models. This could lead to more capable AI systems that can understand and interact with the world more effectively, bringing us closer to achieving AGI.
Abstract
The evolution of machine learning has increasingly prioritized the development of powerful models and more scalable supervision signals. However, the emergence of foundation models presents significant challenges in providing effective supervision signals necessary for further enhancing their capabilities. Consequently, there is an urgent need to explore novel supervision signals and technical approaches. In this paper, we propose verifier engineering, a novel post-training paradigm specifically designed for the era of foundation models. The core of verifier engineering involves leveraging a suite of automated verifiers to perform verification tasks and deliver meaningful feedback to foundation models. We systematically categorize the verifier engineering process into three essential stages: search, verify, and feedback, and provide a comprehensive review of state-of-the-art research developments within each stage. We believe that verifier engineering constitutes a fundamental pathway toward achieving Artificial General Intelligence.