Adam's Law: Textual Frequency Law on Large Language Models

Summary

This paper explores how often words and phrases appear in text and how that relates to making Large Language Models, or LLMs, work better. It proposes that LLMs perform better when they're exposed to more commonly used language, both when given instructions and when being trained.

What's the problem?

LLMs are powerful, but we don't fully understand how the frequency of words and phrases in their training data affects their performance. Most LLMs are trained on secret datasets, so it's hard to know what language they've seen and how often. This makes it difficult to improve them systematically, because we can't easily control the language they learn from.

What's the solution?

The researchers developed a three-part system. First, they suggested a 'Textual Frequency Law' – the idea that LLMs should prefer common language. Since training data is hidden, they figured out a way to estimate how often sentences appear online. Then, they used a tool to rewrite input questions or prompts into more common phrasing. Finally, they trained LLMs by gradually increasing the complexity of the language they were exposed to, starting with the most frequent sentences and moving to less common ones. They tested this on a dataset they created, covering tasks like math, translation, and reasoning.

Why it matters?

This research is important because it suggests a new way to improve LLMs without needing access to their original training data. By focusing on language frequency, we can potentially make these models more reliable and effective, even as they become more complex. It provides a practical approach to optimizing LLM performance using readily available resources and a clever training strategy.