Mark Zuckerberg on:

- Llama 3

- open sourcing towards AGI

- custom silicon, synthetic data, & energy constraints on scaling

- Caesar Augustus, intelligence explosion, bioweapons, $10b models, & much more

Enjoy!

Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Human edited transcript with helpful links here.

Timestamps

(00:00:00) - Llama 3

(00:08:32) - Coding on path to AGI

(00:25:24) - Energy bottlenecks

(00:33:20) - Is AI the most important technology ever?

(00:37:21) - Dangers of open source

(00:53:57) - Caesar Augustus and metaverse

(01:04:53) - Open sourcing the $10b model & custom silicon

(01:15:19) - Zuck as CEO of Google+

Sponsors

If you’re interested in advertising on the podcast, fill out this form.

* This episode is brought to you by Stripe, financial infrastructure for the internet. Millions of companies from Anthropic to Amazon use Stripe to accept payments, automate financial processes and grow their revenue. Learn more at stripe.com.

* V7 Go is a tool to automate multimodal tasks using GenAI, reliably and at scale. Use code DWARKESH20 for 20% off on the pro plan. Learn more here.

* CommandBar is an AI user assistant that any software product can embed to non-annoyingly assist, support, and unleash their users. Used by forward-thinking CX, product, growth, and marketing teams. Learn more at commandbar.com.

Had so much fun chatting with my good friends Trenton Bricken and Sholto Douglas on the podcast.

No way to summarize it, except: 

This is the best context dump out there on how LLMs are trained, what capabilities they're likely to soon have, and what exactly is going on inside them.

You would be shocked how much of what I know about this field, I've learned just from talking with them.

To the extent that you've enjoyed my other AI interviews, now you know why.

So excited to put this out. Enjoy! I certainly did :)

Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. 

There's a transcript with links to all the papers the boys were throwing down - may help you follow along.

Follow Trenton and Sholto on Twitter.

Timestamps

(00:00:00) - Long contexts

(00:16:12) - Intelligence is just associations

(00:32:35) - Intelligence explosion & great researchers

(01:06:52) - Superposition & secret communication

(01:22:34) - Agents & true reasoning

(01:34:40) - How Sholto & Trenton got into AI research

(02:07:16) - Are feature spaces the wrong way to think about intelligence?

(02:21:12) - Will interp actually work on superhuman models

(02:45:05) - Sholto’s technical challenge for the audience

(03:03:57) - Rapid fire

Here is my episode with Demis Hassabis, CEO of Google DeepMind

We discuss:

* Why scaling is an artform

* Adding search, planning, & AlphaZero type training atop LLMs

* Making sure rogue nations can't steal weights

* The right way to align superhuman AIs and do an intelligence explosion

Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here.

Timestamps

(0:00:00) - Nature of intelligence

(0:05:56) - RL atop LLMs

(0:16:31) - Scaling and alignment

(0:24:13) - Timelines and intelligence explosion

(0:28:42) - Gemini training

(0:35:30) - Governance of superhuman AIs

(0:40:42) - Safety, open source, and security of weights

(0:47:00) - Multimodal and further progress

(0:54:18) - Inside Google DeepMind

We discuss:

* what it takes to process $1 trillion/year

* how to build multi-decade APIs, companies, and relationships

* what's next for Stripe (increasing the GDP of the internet is quite an open ended prompt, and the Collison brothers are just getting started).

Plus the amazing stuff they're doing at Arc Institute, the financial infrastructure for AI agents, playing devil's advocate against progress studies, and much more.

Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.

Timestamps

(00:00:00) - Advice for 20-30 year olds

(00:12:12) - Progress studies

(00:22:21) - Arc Institute

(00:34:27) - AI & Fast Grants

(00:43:46) - Stripe history

(00:55:44) - Stripe Climate

(01:01:39) - Beauty & APIs

(01:11:51) - Financial innards

(01:28:16) - Stripe culture & future

(01:41:56) - Virtues of big businesses

(01:51:41) - John

It was a great pleasure speaking with Tyler Cowen for the 3rd time.

We discussed GOAT: Who is the Greatest Economist of all Time and Why Does it Matter?, especially in the context of how the insights of Hayek, Keynes, Smith, and other great economists help us make sense of AI, growth, animal spirits, prediction markets, alignment, central planning, and much more.

The topics covered in this episode are too many to summarize. Hope you enjoy!

Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.

Timestamps

(0:00:00) - John Maynard Keynes

(00:17:16) - Controversy

(00:25:02) - Fredrick von Hayek

(00:47:41) - John Stuart Mill

(00:52:41) - Adam Smith

(00:58:31) - Coase, Schelling, & George

(01:08:07) - Anarchy

(01:13:16) - Cheap WMDs

(01:23:18) - Technocracy & political philosophy

(01:34:16) - AI & Scaling

This is a narration of my blog post, Lessons from The Years of Lyndon Johnson by Robert Caro.

You read the full post here: https://www.dwarkeshpatel.com/p/lyndon-johnson

Listen on Apple Podcasts, Spotify, or any other podcast platform. Follow me on Twitter for updates on future posts and episodes.

This is a narration of my blog post, Will scaling work?.

You read the full post here: https://www.dwarkeshpatel.com/p/will-scaling-work

Listen on Apple Podcasts, Spotify, or any other podcast platform. Follow me on Twitter for updates on future posts and episodes.

A true honor to speak with Jung Chang.

She is the author of Wild Swans: Three Daughters of China (sold 15+ million copies worldwide) and Mao: The Unknown Story.

We discuss:

- what it was like growing up during the Cultural Revolution as the daughter of a denounced official

- why the CCP continues to worship the biggest mass murderer in human history.

- how exactly Communist totalitarianism was able to subjugate a billion people

- why Chinese leaders like Xi and Deng who suffered from the Cultural Revolution don't condemn Mao

- how Mao starved and killed 40 million people during The Great Leap Forward in order to exchange food for Soviet weapons

Wild Swans is the most moving book I've ever read. It was a real privilege to speak with its author.

Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.

Timestamps

(00:00:00) - Growing up during Cultural Revolution

(00:15:58) - Could officials have overthrown Mao?

(00:34:09) - Great Leap Forward

(00:48:12) - Modern support of Mao

(01:03:24) - Life as peasant

(01:21:30) - Psychology of communist society

Andrew Roberts is the world's best biographer and one of the leading historians of our time.

We discussed

* Churchill the applied historian,

* Napoleon the startup founder,

* why Nazi ideology cost Hitler WW2,

* drones, reconnaissance, and other aspects of the future of war,

* Iraq, Afghanistan, Korea, Ukraine, & Taiwan.

Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.

Timestamps

(00:00:00) - Post WW2 conflicts

(00:10:57) - Ukraine

(00:16:33) - How Truman Prevented Nuclear War

(00:22:49) - Taiwan

(00:27:15) - Churchill

(00:35:11) - Gaza & future wars

(00:39:05) - Could Hitler have won WW2?

(00:48:00) - Surprise attacks

(00:59:33) - Napoleon and startup founders

(01:14:06) - Robert’s insane productivity

Here is my interview with Dominic Cummings on why Western governments are so dangerously broken, and how to fix them before an even more catastrophic crisis.

Dominic was Chief Advisor to the Prime Minister during COVID, and before that, director of Vote Leave (which masterminded the 2016 Brexit referendum).

Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.

Timestamps

(00:00:00) - One day in COVID…

(00:08:26) - Why is government broken?

(00:29:10) - Civil service

(00:38:27) - Opportunity wasted?

(00:49:35) - Rishi Sunak and Number 10 vs 11

(00:55:13) - Cyber, nuclear, bio risks

(01:02:04) - Intelligence & defense agencies

(01:23:32) - Bismarck & Lee Kuan Yew

(01:37:46) - How to fix the government?

(01:56:43) - Taiwan

(02:00:10) - Russia

(02:07:12) - Bismarck’s career as an example of AI (mis)alignment

(02:17:37) - Odyssean education

Paul Christiano is the world’s leading AI safety researcher. My full episode with him is out!

We discuss:

- Does he regret inventing RLHF, and is alignment necessarily dual-use?

- Why he has relatively modest timelines (40% by 2040, 15% by 2030),

- What do we want post-AGI world to look like (do we want to keep gods enslaved forever)?

- Why he’s leading the push to get to labs develop responsible scaling policies, and what it would take to prevent an AI coup or bioweapon,

- His current research into a new proof system, and how this could solve alignment by explaining model's behavior

- and much more.

Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.

Open Philanthropy

Open Philanthropy is currently hiring for twenty-two different roles to reduce catastrophic risks from fast-moving advances in AI and biotechnology, including grantmaking, research, and operations.

For more information and to apply, please see the application: https://www.openphilanthropy.org/research/new-roles-on-our-gcr-team/

The deadline to apply is November 9th; make sure to check out those roles before they close.

Timestamps

(00:00:00) - What do we want post-AGI world to look like?

(00:24:25) - Timelines

(00:45:28) - Evolution vs gradient descent

(00:54:53) - Misalignment and takeover

(01:17:23) - Is alignment dual-use?

(01:31:38) - Responsible scaling policies

(01:58:25) - Paul’s alignment research

(02:35:01) - Will this revolutionize theoretical CS and math?

(02:46:11) - How Paul invented RLHF

(02:55:10) - Disagreements with Carl Shulman

(03:01:53) - Long TSMC but not NVIDIA