DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation
Jianzong Wu, Chao Tang, Jingbo Wang, Yanhong Zeng, Xiangtai Li, Yunhai Tong
2024-12-11
Summary
This paper talks about DiffSensei, a new system designed to create customized manga by combining text descriptions with advanced image generation techniques, allowing for better control over characters and scenes.
What's the problem?
While existing models can generate images from text, they often struggle with creating detailed manga that includes multiple characters and their interactions. This lack of control can lead to inconsistencies in character appearances and the overall flow of the story.
What's the solution?
The authors introduce DiffSensei, which uses a combination of a diffusion-based image generator and a multimodal large language model (MLLM). This system allows for precise control over character features and scene layouts. By using a special dataset called MangaZero, which includes thousands of manga pages and character interactions, DiffSensei can adapt characters' expressions and actions based on the text provided. This makes it easier to create coherent and visually engaging manga panels that match the narrative.
Why it matters?
This research is important because it advances the field of AI-generated art, specifically in manga creation. By enabling more dynamic and customizable character interactions, DiffSensei can help artists and writers produce high-quality manga more efficiently. This innovation not only enhances storytelling but also opens up new possibilities for creative expression in digital media.
Abstract
Story visualization, the task of creating visual narratives from textual descriptions, has seen progress with text-to-image generation models. However, these models often lack effective control over character appearances and interactions, particularly in multi-character scenes. To address these limitations, we propose a new task: customized manga generation and introduce DiffSensei, an innovative framework specifically designed for generating manga with dynamic multi-character control. DiffSensei integrates a diffusion-based image generator with a multimodal large language model (MLLM) that acts as a text-compatible identity adapter. Our approach employs masked cross-attention to seamlessly incorporate character features, enabling precise layout control without direct pixel transfer. Additionally, the MLLM-based adapter adjusts character features to align with panel-specific text cues, allowing flexible adjustments in character expressions, poses, and actions. We also introduce MangaZero, a large-scale dataset tailored to this task, containing 43,264 manga pages and 427,147 annotated panels, supporting the visualization of varied character interactions and movements across sequential frames. Extensive experiments demonstrate that DiffSensei outperforms existing models, marking a significant advancement in manga generation by enabling text-adaptable character customization. The project page is https://jianzongwu.github.io/projects/diffsensei/.