Intelligence at the Edge of Chaos
Shiyang Zhang, Aakash Patel, Syed A Rizvi, Nianchen Liu, Sizhuang He, Amin Karbasi, Emanuele Zappala, David van Dijk
2024-10-04
Summary
This paper explores how intelligent behavior emerges in artificial systems, specifically by studying simple rule-based systems called elementary cellular automata (ECA) and their impact on the performance of large language models (LLMs).
What's the problem?
Although large multimodal models (LMMs) have made progress in understanding complex tasks, there is still a question about whether they can truly exhibit intelligent behavior when faced with different types of complexity. The authors aim to investigate how the complexity of rules in systems like ECAs affects the intelligence of models trained to predict these rules. They found that some models struggle with reasoning tasks when trained on overly simple or chaotic systems.
What's the solution?
To address this, the authors trained different LLMs on various ECAs with differing complexities. They discovered that models trained on more complex rules performed better on reasoning tasks, such as predicting chess moves. However, they also noted that too much complexity or chaotic behavior led to poorer performance. This suggests that there is an 'edge of chaos' where the right amount of complexity fosters better intelligence in models.
Why it matters?
This research is important because it helps us understand how exposure to complexity can lead to improved reasoning abilities in AI systems. By identifying the optimal level of complexity for training, this work provides insights into developing more intelligent artificial systems that can handle a wide range of tasks effectively.
Abstract
We explore the emergence of intelligent behavior in artificial systems by investigating how the complexity of rule-based systems influences the capabilities of models trained to predict these rules. Our study focuses on elementary cellular automata (ECA), simple yet powerful one-dimensional systems that generate behaviors ranging from trivial to highly complex. By training distinct Large Language Models (LLMs) on different ECAs, we evaluated the relationship between the complexity of the rules' behavior and the intelligence exhibited by the LLMs, as reflected in their performance on downstream tasks. Our findings reveal that rules with higher complexity lead to models exhibiting greater intelligence, as demonstrated by their performance on reasoning and chess move prediction tasks. Both uniform and periodic systems, and often also highly chaotic systems, resulted in poorer downstream performance, highlighting a sweet spot of complexity conducive to intelligence. We conjecture that intelligence arises from the ability to predict complexity and that creating intelligence may require only exposure to complexity.