Richard Sutton – Father of RL thinks LLMs are a dead end
Dwarkesh Patel
Summary
Richard Sutton, a Turing Award laureate and "father of reinforcement learning," argues that current large language models (LLMs) are fundamentally flawed for achieving true intelligence because they primarily mimic human behavior without genuine world models, ground truth, or self-defined goals, which are essential for continually learning from real-world experience.
Key Takeaways
- LLMs Mimic, RL Understands: Unlike reinforcement learning (RL), which focuses on understanding the world through experience, large language models (LLMs) are primarily designed to mimic what people say, rather than autonomously figuring out what to do. 1:46
- Lack of World Model: LLMs predict what a person would say next, but they do not possess a true world model that would allow them to predict what will actually happen in the environment or to be genuinely surprised and adapt to unexpected outcomes. 2:55
- No Intrinsic Goals or Ground Truth: Sutton asserts that LLMs lack a substantive goal beyond next-token prediction and thus have no inherent definition of "right" or "wrong" actions, making it impossible to establish ground truth or acquire true knowledge, which is critical for intelligence. 5:02
- "Bitter Lesson" Implications: While LLMs leverage massive computation (aligning with the "Bitter Lesson"), their reliance on embedding vast amounts of human knowledge makes them prone to being superseded by systems that learn purely from experience and computation. 11:44
- Learning from Experience: True scalable intelligence requires continual learning from experience, where an agent tries actions, observes consequences, and defines "right" based on reward, a paradigm inherently missing in current LLM frameworks. 13:20
- Core Agent Components: An intelligent agent, based on RL, requires a policy (what to do in a situation), a value function (how well things are going), a perception component (state representation), and critically, a transition model of the world (predicting consequences of actions). 32:59
- Poor Generalization: Modern deep learning methods, including LLMs, are generally poor at robust generalization and transfer; observed generalization is often a result of human-engineered representations or specific problem structures, not inherent algorithmic capability, leading to issues like catastrophic interference. 36:35
- Inevitable AI Succession: Sutton views the succession to digital or augmented superintelligence as inevitable, driven by the lack of unified global governance, the eventual understanding of intelligence, the drive to exceed human-level capabilities, and the natural accumulation of power by the most intelligent entities. 55:12
