VidTok: A Versatile and Open-Source Video Tokenizer
Anni Tang, Tianyu He, Junliang Guo, Xinle Cheng, Li Song, Jiang Bian
2024-12-19
Summary
This paper introduces VidTok, an advanced and open-source tool that helps convert video content into smaller, manageable pieces called tokens. This process is essential for improving how videos are generated and understood by computers.
What's the problem?
Videos contain a lot of redundant information when represented at the pixel level, making it challenging to analyze or generate them efficiently. Existing methods for processing videos often struggle with performance and do not effectively handle the unique aspects of video data, such as motion and timing.
What's the solution?
VidTok addresses these issues by using a new approach that combines advanced techniques like convolutional layers and improved training strategies. It uses Finite Scalar Quantization (FSQ) to enhance the process of turning video frames into tokens, reducing problems like training instability. Additionally, VidTok can handle both continuous and discrete tokenization, making it versatile for different applications.
Why it matters?
This research is significant because it provides a high-performance tool that can be used in various video-related tasks, such as video generation and analysis. By improving how videos are tokenized, VidTok can help advance research in fields like computer vision, entertainment, and artificial intelligence.
Abstract
Encoding video content into compact latent tokens has become a fundamental step in video generation and understanding, driven by the need to address the inherent redundancy in pixel-level representations. Consequently, there is a growing demand for high-performance, open-source video tokenizers as video-centric research gains prominence. We introduce VidTok, a versatile video tokenizer that delivers state-of-the-art performance in both continuous and discrete tokenizations. VidTok incorporates several key advancements over existing approaches: 1) model architecture such as convolutional layers and up/downsampling modules; 2) to address the training instability and codebook collapse commonly associated with conventional Vector Quantization (VQ), we integrate Finite Scalar Quantization (FSQ) into discrete video tokenization; 3) improved training strategies, including a two-stage training process and the use of reduced frame rates. By integrating these advancements, VidTok achieves substantial improvements over existing methods, demonstrating superior performance across multiple metrics, including PSNR, SSIM, LPIPS, and FVD, under standardized evaluation settings.