Dylan Patel — Deep Dive on the 3 Big Bottlenecks to Scaling AI Compute
3/13/20262 hr 31 min
Dylan Patel, founder of SemiAnalysis, provides a deep dive into the 3 big bottlenecks to scaling AI compute: logic, memory, and power.
And walks through the economics of labs, hyperscalers, foundries, and fab equipment manufacturers.
Learned a ton about every single level of the stack. Enjoy!
Watch on YouTube; read the transcript.
Sponsors
* Mercury has already saved me a bunch of time this tax season. Last year, I used Mercury to request W-9s from all the contractors I worked with. Then, when it came time to issue 1099s this year, I literally just clicked a button and Mercury sent them out. Learn more at mercury.com.
* Labelbox noticed that even when voice models appear to take interruptions in stride, their performance degrades. To figure out why, they built a new evaluation pipeline called EchoChain. EchoChain diagnoses voice models’ specific failure modes, letting you understand what your model needs to truly handle interruptions. Check it out at labelbox.com/dwarkesh.
* Jane Street is basically a research lab with a trading desk attached – and their infrastructure backs this up. They’ve got tens of thousands of GPUs, hundreds of thousands of CPU cores, and exabytes of storage. This is what it takes to find subtle signals hidden deep within noisy market data. If this sounds interesting, you can explore open positions at janestreet.com/dwarkesh.
Timestamps
(00:00:00) – Why an H100 is worth more today than 3 years ago
(00:24:52) – Nvidia secured TSMC allocation early; Google is getting squeezed
(00:34:34) – ASML will be the #1 constraint for AI compute scaling by 2030
(00:56:06) – Can’t we just use TSMC’s older fabs?
(01:05:56) – When will China outscale the West in semis?
(01:16:20) – The enormous incoming memory crunch
(01:42:53) – Scaling power in the US will not be a problem
(01:55:03) – Space GPUs aren’t happening this decade
(02:14:26) – Why aren’t more hedge funds making the AGI trade?
(02:18:49) – Will TSMC kick Apple out from N2?
(02:24:35) – Robots and Taiwan risk
Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe
Clips
Showing 10 of 11Transcript preview
First 90 secondsDwarkesh Patel· Host0:00
All right, this is the episode where my roommate teaches me semiconductors.
Dylan Patel· Guest0:02
[laughs] [laughs] It's also the send-off for this, uh, this current set.
Dwarkesh Patel· Host0:06
It's fi-- Yeah, you're-- You know, after you use it, I'm like, "I can't use this again."
Dylan Patel· Guest0:09
[laughs] [laughs] Oh.
Dwarkesh Patel· Host0:10
I gotta get out of here.
Dylan Patel· Guest0:11
No, no sloppy seconds for Dorketh.
Dwarkesh Patel· Host0:12
Yeah. [laughs] [laughs] Okay, Dylan is the, uh, CEO of SemiAnalysis. Dylan, the burning question I have for you, um, if you add up the big four, Amazon, Meta, Google, Microsoft, their combined, uh, forecasted CapEx that y-you published recently this year is six hundred billion dollars. And given, uh, y-you know, yearly prices of renting that compute, that would be, like, close to fifty gigawatts. Now, obviously, we're not putting on fifty gigawatts this year, so presumably that's paying for compute that is gonna be coming online over the coming years. So I have a question about w-what-- how to think about the timeline ar-a-around when that CapEx comes online. Similar question for the labs, where, you know, OpenAI just announced that they raised a hundred and ten billion dollars. Anthropic just announced they raised thirty billion dollars. And if you look at the compute that they have coming online this year, um, you, you should tell me how much it is, but, like, is it not, is it in another four gigawatts total that they'll have this year? It feels like the cost to rent the compute that OpenAI and Anthropic will have this year to, like, sustain their compute spend at, you know, ten, thirteen billion dollars a gigawatt, th-those individual raises alone are, like, w-enough to cover their compute spend for the year, and then this is not even including the revenue that they're gonna earn this year. So help me understand first, when is the timescale