MotiF: Making Text Count in Image Animation with Motion Focal Loss
Shijie Wang, Samaneh Azadi, Rohit Girdhar, Saketh Rambhatla, Chen Sun, Xi Yin
2024-12-25
Summary
This paper talks about MotiF, a new method for generating videos from images based on text descriptions, which improves how well the generated videos match the text, especially regarding motion.
What's the problem?
Current methods for creating videos from images and text often struggle to accurately reflect the details in the text prompts, particularly when it comes to showing motion. This can result in videos that don't look realistic or don't follow the instructions given in the text.
What's the solution?
To solve this problem, the authors introduced MotiF, which focuses on areas of the image that require more motion. They use a technique called optical flow to create a motion heatmap that helps the model learn where to focus its efforts when generating the video. This method allows for better alignment between the text and the resulting video. Additionally, they created a new dataset called TI2V Bench to evaluate how well their method works compared to others. In tests, MotiF performed significantly better than nine other models, achieving high scores in how well it followed the text and how smoothly it animated motion.
Why it matters?
This research is important because it enhances the ability of AI to create videos that are more accurate and visually appealing based on text descriptions. By improving how AI understands and generates motion in videos, MotiF can lead to better applications in fields like animation, gaming, and content creation, making it easier for creators to bring their ideas to life.
Abstract
Text-Image-to-Video (TI2V) generation aims to generate a video from an image following a text description, which is also referred to as text-guided image animation. Most existing methods struggle to generate videos that align well with the text prompts, particularly when motion is specified. To overcome this limitation, we introduce MotiF, a simple yet effective approach that directs the model's learning to the regions with more motion, thereby improving the text alignment and motion generation. We use optical flow to generate a motion heatmap and weight the loss according to the intensity of the motion. This modified objective leads to noticeable improvements and complements existing methods that utilize motion priors as model inputs. Additionally, due to the lack of a diverse benchmark for evaluating TI2V generation, we propose TI2V Bench, a dataset consists of 320 image-text pairs for robust evaluation. We present a human evaluation protocol that asks the annotators to select an overall preference between two videos followed by their justifications. Through a comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourced models, achieving an average preference of 72%. The TI2V Bench is released in https://wang-sj16.github.io/motif/.