LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment
Yibin Wang, Zhiyu Tan, Junyan Wang, Xiaomeng Yang, Cheng Jin, Hao Li
2024-12-09
Summary
This paper talks about LiFT, a new method for improving text-to-video (T2V) models by using human feedback to make the videos they generate better match what people expect.
What's the problem?
Although text-to-video models have made great progress, they often struggle to create videos that accurately reflect the text descriptions provided by users. This is partly because human preferences are subjective and hard to define in a way that machines can understand.
What's the solution?
To tackle this issue, the authors created a dataset called LiFT-HRA, which includes around 10,000 human ratings along with explanations for those ratings. They then trained a reward model named LiFT-Critic that learns to evaluate how well the generated videos align with human expectations. By using this model, they fine-tuned the T2V system to maximize the quality of the videos produced. In their tests, this improved model outperformed existing models on various metrics.
Why it matters?
This research is important because it shows how incorporating human feedback can significantly enhance the performance of AI models, particularly in generating content that meets user expectations. By improving text-to-video generation, LiFT can lead to better applications in entertainment, education, and other fields where visual storytelling is essential.
Abstract
Recent advancements in text-to-video (T2V) generative models have shown impressive capabilities. However, these models are still inadequate in aligning synthesized videos with human preferences (e.g., accurately reflecting text descriptions), which is particularly difficult to address, as human preferences are inherently subjective and challenging to formalize as objective functions. Therefore, this paper proposes LiFT, a novel fine-tuning method leveraging human feedback for T2V model alignment. Specifically, we first construct a Human Rating Annotation dataset, LiFT-HRA, consisting of approximately 10k human annotations, each including a score and its corresponding rationale. Based on this, we train a reward model LiFT-Critic to learn reward function effectively, which serves as a proxy for human judgment, measuring the alignment between given videos and human expectations. Lastly, we leverage the learned reward function to align the T2V model by maximizing the reward-weighted likelihood. As a case study, we apply our pipeline to CogVideoX-2B, showing that the fine-tuned model outperforms the CogVideoX-5B across all 16 metrics, highlighting the potential of human feedback in improving the alignment and quality of synthesized videos.