Semantic IDs for Joint Generative Search and Recommendation
Gustavo Penha, Edoardo D'Amico, Marco De Nadai, Enrico Palumbo, Alexandre Tamborrino, Ali Vardasbi, Max Lefarov, Shawn Lin, Timothy Heath, Francesco Fabbri, Hugues Bouchard
2025-08-20
Summary
This paper investigates how to create item representations called Semantic IDs that work well for both recommending items to users and finding specific items when using a single, powerful AI model, like a Large Language Model.
What's the problem?
When you have an AI model that's good at recommending stuff and also good at searching for specific things, figuring out the best way to represent each item so the AI understands it for both jobs is tricky. If you make the item representation (like a special code) just for recommending, it might not be good for searching, and vice versa. The problem is finding a way to create these item representations that are good for both recommendation and search at the same time within one AI system.
What's the solution?
The researchers tried out different ways to build these Semantic IDs. They explored using separate representations for each task versus creating a single, shared set of representations. They found that using a two-part AI model that first learns general item characteristics from both recommendation and search data, and then uses that information to create a unified set of Semantic IDs, offers a good balance and performs well on both tasks.
Why it matters?
This research is important because it could lead to more efficient and versatile AI systems. Imagine one AI that's great at suggesting movies you'll like and also helps you quickly find exactly the movie you're looking for. By creating these smart, flexible item identifiers, we can build better, more unified AI assistants for everyday tasks.
Abstract
Generative models powered by Large Language Models (LLMs) are emerging as a unified solution for powering both recommendation and search tasks. A key design choice in these models is how to represent items, traditionally through unique identifiers (IDs) and more recently with Semantic IDs composed of discrete codes, obtained from embeddings. While task-specific embedding models can improve performance for individual tasks, they may not generalize well in a joint setting. In this paper, we explore how to construct Semantic IDs that perform well both in search and recommendation when using a unified model. We compare a range of strategies to construct Semantic IDs, looking into task-specific and cross-tasks approaches, and also whether each task should have its own semantic ID tokens in a joint search and recommendation generative model. Our results show that using a bi-encoder model fine-tuned on both search and recommendation tasks to obtain item embeddings, followed by the construction of a unified Semantic ID space provides an effective trade-off, enabling strong performance in both tasks. We hope these findings spark follow-up work on generalisable, semantically grounded ID schemes and inform the next wave of unified generative recommender architectures.