LLM Pretraining with Continuous Concepts
Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuandong Tian, Jason Weston, Xian Li
2025-02-13
Summary
This paper talks about CineMaster, a new AI system that can create videos from text descriptions while giving users the ability to control the 3D aspects of the video, like where objects are placed and how the camera moves, similar to how a film director would.
What's the problem?
Current AI systems that make videos from text descriptions don't give users much control over the 3D aspects of the scene, like where things are placed or how the camera moves. This makes it hard for people to create exactly the video they want, especially if they need specific camera angles or object placements.
What's the solution?
The researchers created CineMaster, which works in two stages. First, it lets users place objects in a 3D space and plan camera movements, kind of like setting up a virtual movie set. Then, it uses this information along with the text description to generate a video that matches what the user wanted. They also created a way to automatically label lots of existing videos with 3D information, which helps train the AI to understand 3D scenes better.
Why it matters?
This matters because it could change how people create videos using AI. Instead of just getting a random video based on a text description, users could have much more control over the final product, almost like being a movie director. This could be useful for filmmakers, advertisers, or anyone who needs to create specific video content without expensive equipment or large production teams.
Abstract
Next token prediction has been the standard training objective used in large language model pretraining. Representations are learned as a result of optimizing for token-level perplexity. We propose Continuous Concept Mixing (CoCoMix), a novel pretraining framework that combines discrete next token prediction with continuous concepts. Specifically, CoCoMix predicts continuous concepts learned from a pretrained sparse autoencoder and mixes them into the model's hidden state by interleaving with token hidden representations. Through experiments on multiple benchmarks, including language modeling and downstream reasoning tasks, we show that CoCoMix is more sample efficient and consistently outperforms standard next token prediction, knowledge distillation and inserting pause tokens. We find that combining both concept learning and interleaving in an end-to-end framework is critical to performance gains. Furthermore, CoCoMix enhances interpretability and steerability by allowing direct inspection and modification of the predicted concept, offering a transparent way to guide the model's internal reasoning process.