ColorFlow: Retrieval-Augmented Image Sequence Colorization
Junhao Zhuang, Xuan Ju, Zhaoyang Zhang, Yong Liu, Shiyi Zhang, Chun Yuan, Ying Shan
2024-12-17
Summary
This paper discusses ColorFlow, a new system designed to automatically add color to black-and-white image sequences, like comics or cartoons, while keeping the characters and objects consistent throughout the images.
What's the problem?
Colorizing black-and-white images is a complex task, especially when it comes to maintaining the identity of characters and objects across multiple frames. Current methods struggle with making large changes and often require complicated setups that are not practical for industrial use, making it hard to achieve consistent and high-quality results.
What's the solution?
ColorFlow introduces a three-stage framework that uses a combination of advanced techniques to colorize images effectively. It utilizes a Retrieval-Augmented Pipeline to find relevant color references from a pool of images. The system is designed with two branches: one for extracting color identities and another for the actual colorization process. This approach allows for better control over how colors are applied while maintaining the integrity of the original characters and objects. The framework also includes a benchmark called ColorFlow-Bench to evaluate its performance against other methods.
Why it matters?
The ColorFlow system is significant because it sets a new standard in the field of image colorization, particularly for industries like animation and comics where consistency is crucial. By improving how colors are applied to image sequences, it can help artists and creators produce high-quality work more efficiently, ultimately benefiting the art industry as a whole.
Abstract
Automatic black-and-white image sequence colorization while preserving character and object identity (ID) is a complex task with significant market demand, such as in cartoon or comic series colorization. Despite advancements in visual colorization using large-scale generative models like diffusion models, challenges with controllability and identity consistency persist, making current solutions unsuitable for industrial application.To address this, we propose ColorFlow, a three-stage diffusion-based framework tailored for image sequence colorization in industrial applications. Unlike existing methods that require per-ID finetuning or explicit ID embedding extraction, we propose a novel robust and generalizable Retrieval Augmented Colorization pipeline for colorizing images with relevant color references. Our pipeline also features a dual-branch design: one branch for color identity extraction and the other for colorization, leveraging the strengths of diffusion models. We utilize the self-attention mechanism in diffusion models for strong in-context learning and color identity matching. To evaluate our model, we introduce ColorFlow-Bench, a comprehensive benchmark for reference-based colorization. Results show that ColorFlow outperforms existing models across multiple metrics, setting a new standard in sequential image colorization and potentially benefiting the art industry. We release our codes and models on our project page: https://zhuang2002.github.io/ColorFlow/.