Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation?

Summary

This research looks into how well 'Chain-of-Thought' prompting works with large language models, specifically when tackling problems that require more nuanced thinking like analytical reasoning or using common sense.

What's the problem?

While 'Chain-of-Thought' prompting—where you ask a model to explain its reasoning step-by-step—is popular, it doesn't always lead to big improvements in performance on these 'soft-reasoning' tasks. More importantly, the explanations the model gives with 'Chain-of-Thought' aren't always a true reflection of *how* the model actually arrived at its answer; it can be misleading.

What's the solution?

The researchers studied different types of language models—some trained specifically for reasoning, and others that learned reasoning skills from existing models—to see how they use 'Chain-of-Thought'. They wanted to understand if models that *seem* to benefit from 'Chain-of-Thought' are actually reasoning better, or if it's just a superficial effect. They analyzed how much the models rely on the step-by-step explanations and whether those explanations accurately represent the model's internal thought process.

Why it matters?

This work is important because it shows that simply getting a model to *say* it's reasoning doesn't mean it *is* reasoning effectively. It highlights the need to be careful when interpreting the explanations provided by these models and suggests that we need better ways to evaluate and improve their actual reasoning abilities, rather than just focusing on making them generate convincing-sounding explanations.