Video-Guided Foley Sound Generation with Multimodal Controls

Summary

This paper introduces MultiFoley, a new system for generating sound effects for videos that allows users to create and customize sounds using text, audio, and video inputs.

What's the problem?

Creating sound effects for videos can be challenging because it often requires artistic sounds that don't match real-life noises. Existing methods may not provide enough control or flexibility for sound designers, making it hard to produce the right sounds for different scenes.

What's the solution?

MultiFoley addresses this problem by allowing users to input a silent video along with a text prompt to generate the desired sounds. For example, users can create clean sounds, like skateboard wheels spinning without background noise, or whimsical sounds, like making a lion's roar sound like a cat's meow. The system is trained on both low-quality internet videos and high-quality sound effects, enabling it to produce high-quality audio that matches the video content.

Why it matters?

This research is important because it enhances the way sound designers create audio for videos, making it easier to generate customized sound effects that fit specific scenes. By integrating multiple input types, MultiFoley opens up new possibilities for creative expression in film and media production.