I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm
Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, Wenhao Huang, Jiajun Zhang
2024-08-16
Summary
This paper introduces I-SHEEP, a new method for training large language models (LLMs) that allows them to continuously improve their understanding and performance without needing initial data.
What's the problem?
Most current methods for training LLMs treat them as passive tools that only learn from the data they are given. This means they miss out on the opportunity to learn actively and adapt over time, which can limit their effectiveness and understanding.
What's the solution?
I-SHEEP changes this by allowing LLMs to self-align and improve continuously from scratch. It uses an iterative process that helps models learn more like humans do, leading to significant improvements in performance on various tasks. The method has shown impressive results, with substantial gains in accuracy on different benchmarks compared to previous models.
Why it matters?
This research is important because it opens up new possibilities for making LLMs smarter and more adaptable. By enabling these models to learn continuously and improve on their own, we can enhance their usefulness in applications like education, customer service, and content creation.
Abstract
Large Language Models (LLMs) have achieved significant advancements, however, the common learning paradigm treats LLMs as passive information repositories, neglecting their potential for active learning and alignment. Some approaches train LLMs using their own generated synthetic data, exploring the possibility of active alignment. However, there is still a huge gap between these one-time alignment methods and the continuous automatic alignment of humans. In this paper, we introduce I-SHEEP, an Iterative Self-EnHancEmEnt Paradigm.This human-like paradigm enables LLMs to continuously self-align from scratch with nothing. Compared to the one-time alignment method Dromedary sun2023principledriven, which refers to the first iteration in this paper, I-SHEEP can significantly enhance capacities on both Qwen and Llama models. I-SHEEP achieves a maximum relative improvement of 78.2\% in the Alpaca Eval, 24.0\% in the MT Bench, and an absolute increase of 8.88\% in the IFEval accuracy over subsequent iterations in Qwen-1.5 72B model. Additionally, I-SHEEP surpasses the base model in various standard benchmark generation tasks, achieving an average improvement of 24.77\% in code generation tasks, 12.04\% in TrivialQA, and 20.29\% in SQuAD. We also provide new insights based on the experiment results. Our codes, datasets, and models are available at https://anonymous.4open.science/r/I-SHEEP.