Eric Jang – Building AlphaGo from scratch
5/15/20262 hr 37 min
Eric Jang walks through how to build AlphaGo from scratch, but with modern AI tools.
Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn.
Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second.
Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside.
Watch on YouTube. Read the transcript.
And check out the flashcards I wrote to retain the insights.
Sponsors
* Cursor‘s agent SDK let me build a pipeline to generate flashcards for this episode. For each card, I had an agent read the transcript, ingest blackboard screenshots, generate an SVG visual, and run everything through a critic. A durable agent is much better at this kind of work than a chain of LLM calls, and Cursor’s SDK made it easy. Check out the cards at flashcards.dwarkesh.com and get started with the SDK at cursor.com/dwarkesh
* Jane Street gave me a real deep-dive tour of one of their datacenters. I got to ask a bunch of questions to Ron Minsky, who co-leads Jane Street’s tech group, and Dan Pontecorvo, who runs Jane Street’s physical engineering team. They were willing to literally pull up the floorboards and take out racks to explain how everything works. Check out the full tour at janestreet.com/dwarkesh
Timestamps
(00:00:00) – Basics of Go
(00:08:17) – Monte Carlo Tree Search
(00:32:04) – What the neural network does
(01:00:33) – Self-play
(01:25:38) – Alternative RL approaches
(01:45:47) – Why doesn't MCTS work for LLMs
(02:01:09) – Off-policy training
(02:12:02) – RL is even more information inefficient than you thought
(02:22:16) – Automated AI researchers
Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe
Clips
Transcript preview
First 90 secondsDwarkesh Patel· Host0:00
Today I'm here with Eric Jiang, who was most recently vice president of AI at 1X Technologies, before that senior research scientist at what is now Google DeepMind Robotics, and you've been on sabbatical for the last few months. One of the things you've been doing is rebuilding, and improving, and hacking on AlphaGo. And so we're, today what we're gonna do is you're gonna explain building AlphaGo from scratch and what it tells us about the future of AI research and development. But, uh, before we get to that, why is AlphaGo interesting? Why is this, why is this the project you decided to do on sabbatical rather than just hanging out at the beach?
Eric Jang· Guest0:33
Sure, yeah. Um, I like making things, and AlphaGo and Go AI is one of those things that really got me into the field. Uh, when I saw the kind of early breakthroughs, um, on AlphaGo in 2014, 2015, 2016 and so forth, it was just profound to see, you know, how smart AI systems could become, and the, the kind of computational complexity class that they could tackle with deep learning. Um, this is a problem that has, you know, long been understood to be kind of intractable for a search, and yet, um, it was solved, um, through, through deep learning. And so, so that was quite mysterious to me, and I've always wanted to understand that phenomena a little bit better. My training is often in deep neural nets for robotics, where it's, uh, the, the decisions made by the neural networks are a bit more intuitive, but AlphaGo is a sort of problem where the, the decisions are actually the result of a very, very deep search. And it's always been very mysterious to me how, like, a 10-layer network can sort of-