Vector-ICL: In-context Learning with Continuous Vector Representations
Yufan Zhuang, Chandan Singh, Liyuan Liu, Jingbo Shang, Jianfeng Gao
2024-10-13
Summary
This paper introduces Vector-ICL, a method that allows large language models (LLMs) to learn from continuous vector representations instead of just text, expanding their ability to understand and process different types of data.
What's the problem?
Traditional in-context learning (ICL) methods for LLMs have primarily focused on text data, which limits their application to other types of information that might not fit well into text format. Many important data types, like sensor readings or numerical data, are better represented as continuous vectors rather than discrete tokens. This can make it challenging for LLMs to effectively learn and apply knowledge from these diverse sources.
What's the solution?
The authors propose a new approach called Vector-ICL, which involves aligning input data with the LLM's embedding space using lightweight projectors. This allows the model to process and learn from continuous vectors effectively. They found that training these projectors with general language modeling objectives helps the LLM understand the continuous data better. The results showed that Vector-ICL outperformed traditional few-shot learning methods across various tasks, including text classification and numerical function regression, demonstrating the model's ability to handle different types of data more effectively.
Why it matters?
This research is significant because it expands the capabilities of LLMs beyond just text, enabling them to work with a wider range of data types. By improving how these models learn from continuous representations, Vector-ICL could lead to advancements in fields like data analysis, scientific research, and any application that requires understanding complex numerical or sensor data.
Abstract
Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities on textual data. We explore whether these capabilities can be extended to continuous vectors from diverse domains, obtained from black-box pretrained encoders. By aligning input data with an LLM's embedding space through lightweight projectors, we observe that LLMs can effectively process and learn from these projected vectors, which we term Vector-ICL. In particular, we find that pretraining projectors with general language modeling objectives enables Vector-ICL, while task-specific finetuning further enhances performance. In our experiments across various tasks and modalities, including text reconstruction, numerical function regression, text classification, summarization, molecule captioning, time-series classification, graph classification, and fMRI decoding, Vector-ICL often surpasses both few-shot ICL and domain-specific model or tuning. We further conduct analyses and case studies, indicating the potential of LLMs to process vector representations beyond traditional token-based paradigms.