BOND: Aligning LLMs with Best-of-N Distillation

Summary

This paper presents BOND, a new algorithm designed to improve the performance of large language models (LLMs) by using a method called Best-of-N Distillation. The goal is to make LLMs generate high-quality responses while reducing the computational resources needed during inference.

What's the problem?

While reinforcement learning from human feedback (RLHF) has helped improve the quality and safety of LLMs, traditional methods like Best-of-N sampling require a lot of computational power because they evaluate multiple candidate responses to choose the best one. This makes it hard to use them efficiently in real-time applications where quick responses are needed.

What's the solution?

BOND addresses this issue by creating a new RLHF algorithm that mimics the Best-of-N sampling process without needing to evaluate multiple candidates at once. Instead, it uses a technique called distribution matching to adjust the model's output to be more like what Best-of-N would produce. By employing Jeffreys divergence, BOND balances how the model covers different possible outputs while also focusing on specific desirable outcomes. This allows BOND to achieve strong performance with only a single output sample during inference, making it much more efficient.

Why it matters?

This research is significant because it enhances the efficiency of LLMs, enabling them to provide high-quality answers quickly and with less computational cost. This improvement can lead to better performance in various applications, such as chatbots, virtual assistants, and any system that relies on fast and accurate text generation.