DreamingComics: A Story Visualization Pipeline via Subject and Layout Customized Generation using Video Models
Patrick Kwon, Chen Chen
2025-12-02
Summary
This paper introduces DreamingComics, a new system for automatically creating comic book-style visualizations from text scripts. It aims to make these visualizations more consistent in terms of character appearance and artistic style, and also to give more control over where characters are placed in each panel.
What's the problem?
Existing methods for turning stories into visuals struggle with two main things. First, they rely heavily on just the text to figure out where to put characters, which isn't always enough. Second, it's hard to make sure characters look the same throughout the comic and that the overall art style remains consistent from panel to panel. Basically, the visuals often feel disjointed and lack a cohesive artistic vision.
What's the solution?
The researchers built upon a powerful AI model originally designed for creating videos. They modified it to understand and maintain consistent character identities and art styles. A key innovation is 'RegionalRoPE,' a technique that tells the AI exactly where to draw each character within a panel, based on a comic-style layout. They also used a special 'masked condition loss' to ensure characters stay within their assigned areas. Finally, they used a large language model to automatically generate these comic-style layouts from the story's text.
Why it matters?
This work is important because it significantly improves the quality and consistency of automatically generated comic visualizations. The results show a noticeable improvement in both how consistently characters are drawn and how well the art style is maintained, making the generated comics more visually appealing and easier to follow. This could have applications in automated content creation, storyboarding, and even personalized comic generation.
Abstract
Current story visualization methods tend to position subjects solely by text and face challenges in maintaining artistic consistency. To address these limitations, we introduce DreamingComics, a layout-aware story visualization framework. We build upon a pretrained video diffusion-transformer (DiT) model, leveraging its spatiotemporal priors to enhance identity and style consistency. For layout-based position control, we propose RegionalRoPE, a region-aware positional encoding scheme that re-indexes embeddings based on the target layout. Additionally, we introduce a masked condition loss to further constrain each subject's visual features to their designated region. To infer layouts from natural language scripts, we integrate an LLM-based layout generator trained to produce comic-style layouts, enabling flexible and controllable layout conditioning. We present a comprehensive evaluation of our approach, showing a 29.2% increase in character consistency and a 36.2% increase in style similarity compared to previous methods, while displaying high spatial accuracy. Our project page is available at https://yj7082126.github.io/dreamingcomics/