Fresh off raising a monster $15B, Marc Andreessen has lived through multiple computing platform shifts firsthand, from Mosaic and Netscape to cofounding A16z.

In this episode, Marc joins swyx and Alessio in a16z’s legendary Sand Hill Road office to argue that AI is not just another hype cycle, but the payoff of an “80-year overnight success”: from neural nets and expert systems to transformers, reasoning models, coding, agents, and recursive self-improvement. He lays out why he thinks this moment is different, why AI is finally escaping the old boom-bust pattern, and why the real bottleneck may be less about models than about the messy institutions, incentives, and social systems that struggle to absorb technological change.

This episode was a dream come true for us, and many thanks to Erik Torenberg for the assist in setting this up. Full episode on YouTube!

We discuss:

* Marc’s long view on AI: from the 1980s AI boom and expert systems to AlexNet, transformers, and why he sees today’s moment as the culmination of decades of compounding technical progress

* Why “this time is different”: the jump from LLMs to reasoning, coding, agents, and recursive self-improvement, and why Marc thinks these breakthroughs make AI real in a way prior cycles were not

* AI winters vs. “80-year overnight success”: why the field repeatedly swings between utopianism and doom, and why Marc thinks the underlying researchers were mostly right even when the timelines were wrong

* Scaling laws, Moore’s Law, and what to build: why he believes AI scaling laws will continue, why the outside world is messier than lab purists assume, and how startups can still create durable value on top of rapidly improving models

* The dot-com crash and AI infrastructure risk: Marc’s comparison between today’s AI capex boom and the fiber/data-center overbuild of 2000, plus why he thinks this cycle is different because the buyers are huge cash-rich incumbents and demand is already here

* Why old NVIDIA chips may be getting more valuable: the pace of software progress, chronic capacity shortages, and the idea that even current models are “sandbagged” by supply constraints

* Open source, edge inference, and the chip bottleneck: why Marc thinks local models, Apple Silicon, privacy, trust, and economics all point toward a major role for edge AI

* American vs. Chinese open source AI: DeepSeek as a “gift to the world,” why open models matter not just because they’re free but because they teach the world how things work, and how open source strategies may shift as the market consolidates

* Why Pi and OpenClaw matter so much: Marc’s claim that the combination of LLM + shell + filesystem + markdown + cron loop is one of the biggest software architecture breakthroughs in decades

* Agents as the new “Unix”: how agent state living in files allows portability across models and runtimes, and why self-modifying agents that can extend themselves may redefine what software even is

* The future of coding and programming languages: why Marc thinks software becomes abundant, why bots may translate freely across languages, and why “programming language” itself may stop being a salient concept

* Browsers, protocols, and human readability: lessons from Mosaic and the web, why text protocols and “view source” mattered, and how similar principles may shape AI-native systems

* Real-world OpenClaw use: health dashboards, sleep monitoring, smart homes, rewriting firmware on robot dogs, and why the most aggressive users are discovering both the power and danger of agents first

* Proof of human vs. proof of bot: why Marc thinks the internet’s bot problem is now unsolvable via detection alone, and why biometric + cryptographic proof of human becomes necessary

Timestamps

* 00:00 Marc on AI’s “80-Year Overnight Success”

* 00:01 A Quick Message From swyx

* 01:44 Inside a16z With Marc Andreessen

* 02:13 The Truth About a16z’s AI Pivot

* 03:29 Why This AI Boom Is Not Like 2016

* 06:33 Marc on AI Winters, Hype Cycles, and What’s Different Now

* 10:09 Reasoning, Coding, Agents, and the New AI Breakthroughs

* 12:13 What Founders Should Build as Models Keep Improving

* 16:33 AI Capex, GPU Shortages, and the Dot-Com Crash Analogy

* 24:54 Open Source AI, Edge Inference, and Why It Matters

* 33:03 Why OpenClaw and PI Could Change Software Forever

* 41:37 Agents, the End of Interfaces, and Software for Bots

* 46:47 Do Programming Languages Even Have a Future?

* 54:19 AI Agents Need Money: Payments, Crypto, and Stablecoins

* 56:59 Proof of Human, Internet Bots, and the Drone Problem

* 01:06:12 AI, Management, and the Return of Founder-Led Companies

* 01:12:23 Why the Real Economy May Resist AI Longer Than Expected

* 01:15:53 Closing Thoughts

Transcript

Marc: Something about AI that causes the people in the field, I would say, to become both excessively utopian and excessively apocalyptic. Having said that, I think what’s actually happened is an enormous amount of technical progress that built up over time. And like for, for example, we now know that neural network is the correct architecture.And I, I will tell you like there was a 60 year run where that was like a, you know, or even 70 years where that was controversial. And so, so the way I think about what’s happening is basically, I think, I think about basically the, the, the period we’re in right now is it’s, I call it 80 year overnight success, right?Which is like, it’s an overnight success ‘cause it’s like bam, you know, chat GPT hits and then, and then oh one hits, and then, you know, open claw hits and like, you know, these are open, these are, these are like overnight, like radical, overnight transformative successes, but they’re drawing on an 80 year sort of wellspring backlog, you know, of, of, of, of ideas and thinking it’s not just that it’s all brand new, it’s that it’s an unlock of all of these decades of like very serious, hardcore research.If I were 18, like this is a hundred, this is what I would be spending all of my time on. This is like such an incredible conceptual breakthrough.swyx: Before we get into today’s episode, I just have a small message for listeners. Thank you. We will not be able to bring you the ai, engineering, science, and entertainment contents that you so clearly want if you didn’t choose to also click in and tune into our content.We’ve been approached by sponsors on an almost daily basis, but fortunately enough of you actually subscribed to us to keep all this sustainable without ads, and we wanna keep it that way. But I just have one favor to ask all of you. The single, most powerful, completely free thing you can do is to click that subscribe button.It’s the only thing I’ll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring the in space to you each and every week. If you do it, I promise you will never stop working to make the show even better. Now, let’s get into it.Alessio: Hey everyone, welcome to the Lidian Space Pockets. This is CIO, founder Kernel Labs, and I’m joined by s Swix, editor of Lidian Space.swyx: Hello. And we’re in a 16 Z with a, uh, mark G and welcome.Marc: Yes, yes. A and what, half of 16? Something like that. A one. Exactly,swyx: exactly. Uh, apparently this is the, the final few days in your, your current office.You’re moving across the road.Marc: Uh, we’re, yeah. We have a, we have some, we have some projects underway, but yeah, this is actually, oh, this is the original. We’re in actually the original office. We’re in the, we’re in the, we’re, we’re in the whole thing.swyx: It’s beautiful. Yeah. Great.Marc: Thank you.swyx: So I have to come out, uh, this is a, you know, I wanted to pick a spicy start in October, 2022.I just made friends with Roone and, uh, I wanted to give him something to sort of be spicy about. And I said, uh. Uh, it’ll never not be funny. The A 16 Z was constantly going. The future is where the smart people choose to spend their time and then going deep into crypto and not in ai. And that was in October 22nd, 2022.And Ruen says there was an internal meeting in a 16 Z to reorient around Gen ai. Obviously you have, but was there a meeting? What, what was that?Marc: I mean, I don’t, look, I’ve been doing AI since the late eighties.swyx: Yeah.Marc: So I, I don’t know, like all that, as far as I’m concerned, this stuff is all Johnny cum lately.Yeah. You, I mean, look, we’ve been doing ar entire existence. I mean, we’ve been doing AI machine learning deep, you know, deeply. We’ve been doing this stuff way from the beginning. Obviously a AI is just core to computer science. I, I, I actually view them as like quite, uh, quite continuous. Um, you know, Ben and I both have computer science degrees.Um, you know, we, we both, Ben, Ben and I actually both are world enough to remember the actual AI boom in the 1980s. Yeah. There was like a, there was a big AI boom at the time. Um, and there was a, was names like expert systems. Um, and they of like lisp and lisp machines. Uh, I, I coded in lisp. I was coding a lisp in 1989.When that was the, the language of the AI future. Um, yeah. So this is something that we’re like completely, you completely comfortable with. I’ve been doing the whole time and are very enthusiastic aboutswyx: is there a strong, like this time is different because, uh, my closest analog was 20 16 17. It was an AI boom.Mm-hmm. And it petered out very, very quickly. Um, we, it just, it just in terms of investingMarc: sort of, sort of,swyx: yeah. Investment, investment excitement.Marc: Although that’s really when the, the, the Nvidia phenomenon really, it was, I would say it was in that period when it was very clear that at, at the time it, the vocabulary was more machine learning, but it, it was very clear at that time that machine learning was hitting some sort of takeoff point.Alessio: Yeah.Marc: Well, and as you guys, you guys have talked about this at length on, on your thing, but, you know, if you really track what happened, I think the real story is, it was, it was the Alex net, uh, basically breakthrough in like 2013. That was the, that was the real knee in the curve. Um, and then it was obviously the transformer breakthrough in 17.Alessio: Yeah.Marc: Um, and then everything that followed. But, but, you know, look, machine learning, you know, there were, you know, look, uh, I mean look, I’ve been working, you know, I’ve been working with, uh, one of my, you know, kind of projects working with Facebook since 2004. Um, and on the board since 2007, and of course, you know, they, they started using machine learning very early, um, and, you know, have used it basically, you know, for like 20 years for, you know, content, you know, feed optimization and advertising optimization.And obviously many, you know, financial services. You know, many, many, many companies, many different sectors have been doing this. And so it’s like one of these things, it’s like, it’s not a, it’s not a single thing. Like it’s, it’s like, it’s like layers, right? Yeah. Um, and, and the layers arrive at different paces and, but they kind of build up.swyx: Yeah.Marc: Uh, they kind of build up over time and then, and then, yeah. And then look, in retrospect, it was 2017 was kind of the, you know, the key, the key point with the trans transformer and then. And then as you guys know, there was this really weird like four year period where it’s like the, the transformer existed and then it was just like,swyx: let’s go.Yeah.Marc: Well, but, but it was just, but, but between 2020, but between 2017 and 2021, I mean, that was the era of which like companies like Google had internal chat Botts, but they weren’t letting anybody use them.swyx: Yeah.Marc: Right. And then, you know, and then OpenAI developed Chat GT or GPT two, and then they told everybody, this is way too dangerous to deploy.Right. Yeah. You know, we can’t possibly let normal people, normal people use this thing. And then you, you guys, I’m sure remember AI Dungeon, um mm-hmm. So the o for, there was like a year where like the only way for a normal person to use GP T three was in, in AI dungeon.Alessio: Yeah.Marc: And so you, you, we would do this, you’d go in there and you’d pretend to play Dungeons and Dragons.In reality, you’re just trying to talk to talk to GPT. And so there was this, you know, there was this long, you know, and I, you know, the big, big companies, you know, big companies are cautious and, you know, the big companies were cautious. It, it, by the way, it took open ai. You know, they, they, they talk about this, it took open AI time to actually adjust, you know, kind of re redirect their researchswyx: path.I, I think, uh, let say Rosewood, right? Uh, the, the dinner that founded OpenAI was right there.Marc: Right, right. But that, that dinner would’ve taken place in 20swyx: 18Marc: 19. The formation of OpenAI Uhhuh as late as 2018.swyx: Uh, uh, sorry. Uh, no, I’m, I’m, I’m, I’m wrong. Probably It should be 20. Yeah. They just celebrated a 10 year anniversary, so it it is 2025.Yeah, so, so 2015?Marc: Yeah. 2015. Yeah. 2015. But then, uh, um, Alec Radford did G PT one in what, probablyswyx: mm-hmm. 17, 18,Marc: yeah. 17, 18. So it, yeah. For, and then, and then they didn’t really, and then GPT three was what? 2020? 2020.swyx: 2020.Marc: Because that became copilot immediately. Even open ai, which has been, you know, the leader of, of this thing in the last decade, you know, e even they had to adapt and, and, and lean into the new thing.And so. Um, yeah, I, I think it’s just this process of basically sort of wave after wave layer after layer, you know, building on itself. And then you kind of get these catalytic moments where, where the whole thing pops and, and obviously that’s what’s happening now.swyx: Is it useful to think about will there be any ai, winter?‘cause there’s always these patterns. Like, is this, in the summer is something I constantly think about because do I get, do I just like. Just get endlessly hyped and just trust that I will only be early and never wrong or right. Well, are we, will there be a winter?Marc: So there’s something about, say the following.There’s something about AI that has led to this repeated pattern. Um, and, and, and you guys know this,swyx: it’s summer, winter, summer,Marc: winter, summer, winter, summer, winter. And it goes back 80 years. Yeah. 80 years. Uh, so the original neural network paper was 1943. Right. Which is, which is amazing. Uh, that it was, it was far back that long.And then there was you, if you guys have ever talked about this on your show, but there was this, uh, there was a big, uh, there was an a GI conference at Dartmouth University in 1950. 55. 55, yeah. And they got a NSF grant to, uh, for the, all the AI experts at the time to spend the summer together. And they figured if they had 10 weeks together, they could get a GI, uh, at the other end.And they got their, by the way, they got the grant, they got the 10 weeks and then, you know, 1955, you know. No, no. A GI. And like I said, I, I lived through the eighties version of this where there was a big, a big boom and a crash. And so, so there is this thing, and there, there is something about AI that causes the people in the field, I would say, to become both excessively utopian and excessively apocalyptic.Um, and, and it’s probably on both sides of like the, the, the boom bus cycle. You, you kind of see that play out. Having said that, I think what’s actually happened is like just, and you know, and we now know in retrospect like an enormous amount of technical progress that built up over time. And like for, for example, we now know that neural network is the correct architecture.And I, I will tell you like there was a 60 year run where that was like a, you know, or even 70 years or that was controversial. And, and we now know that that’s the case. And so we, we now, you know, everything we’re building on today just sort of derives from the original idea in 1943. And so, so in retrospect, we, we now know that like, these, these guys are right.They, they, you know, they would get the timing wrong and they thought, you know, capabilities would arrive faster, or they were, it could be turned into businesses sooner or whatever, but like, they were fundamentally, the, the scientists who worked on this over the course of decades were fundamentally correct about what they were doing.And, and the, and the payoff from, from, from all their work is happening now. And so, so the way I think about what’s happening is basically, I think, I think about basically the, the, the period we’re in right now is it’s, I call it 80 year overnight success, right? Which is like, it’s an overnight success.‘cause it’s like bam, you know, chat, GPT hits and then, and then oh one hits, and then, you know, open claw hits and like, you know, these are open, these are, these are like overnight, like radical, overnight transformative successes, but they’re drawing on an 80 year sort of wellspring backlog, you know, of, of, of, of ideas and thinking it’s not just that it’s all brand new, it’s that it’s an unlock of all of these decades of like very serious, hardcore research.Um, and thinking, and look, there were AI researchers who spent their entire lives. They got their PhD. They, they worked for, they’ve researched for 40 years. They retired in a lot of cases, they passed away and they never actually saw it work.swyx: Yeah. It’s all sad.Marc: It is. It is sad. It’s sad. Knewswyx: Jeff Hinton was like the last guy.Marc: Yeah. Yeah. Well, there were the guys, uh, was a guy, Alan Newell. I mean, there’s tons of John McCarthy. You know, John McCarthy was like one of the inventors in the field. He’s one of the guys who organized the Dartmouth Conference and you know, he taught at Stanford for 40 years. Wow. And passed, you know, passed away, I don’t know, whatever, 10, 10 years ago or something.Never, never actually go. Got to see it happen. But like, it is amazing in retrospect, like, these guys were incredibly smart and they worked really hard and they were correct. So anyway, so then it’s like, okay, you know, say history doesn’t repeat, but it rhymes. It’s like, okay, does that mean that there’s gonna be another, like, you know, basically boom buzz cycle.And I, I will tell you, like, let, like in a sense, like yes, everything goes through cycles and, you know, people get overly enthusiastic and overly depressed and there’s, there’s a time, there’s a timelessness to that. Having said that, there’s just no question. Um, so the form, the foremost dangerous words in investing this time are, this time is different.Do you know the 12 most dangerous words investing? No. The four most d foremost dangerous words in investing are this time is different. Yeah. Um, the 12 most dangerous words. And so like, I’ll tell you what’s different. Like now it’s working like, like there’s just no, I mean, look, there’s just no question.And by the way, I, I’ll just give you guys my take. Like L LLMs, like from, from basically the Chad G PT moment through to spring of 25. I think you could still, I think well intention, well, and of. Form skeptics could still say, oh, this is just pattern completion. And oh, these things don’t really understand what they’re doing.And you know, the hall hallucination rates are way too high. And, you know, this is gonna be great for creative writing and creating, you know, Shakespeare and so sonnets and, you know, as, as rap lyrics or whatever, like, it’s gonna be great and all that stuff, but we’re not gonna be able to harness this to make this relevant in, you know, coding or in medicine or in law or in, you know, you know, kind of feels that, you know, kind of really, really matter.And I think basically it was the reasoning breakthrough. It, it was oh one and then R one that basically answered that question basically said, oh no, we’re gonna be able to actually turn this into something that’s gonna work in the real world. And, and then obviously the coding breakthrough over the, over basically the coding breakthrough that kind of catalyzed over the holiday break was kind of the third step in that.Mm-hmm. Where you’re just like, alright, if, if, you know, if Linus Tova is saying that the AI coding is no better than he is like. Like, that’s, that’s never happened before. That’s theswyx: benchmark.Marc: Yeah. That’s never happened before. And so now we know that it’s, it’s gonna sweep through coding and, and then, and then we, we know, you know, we know that if it’s gonna work in coding, it’s gonna work in everything else.Right. It’s just then, because that’s, that’s like, that’s like, that’s like the hardest in many ways. That’s the hardest example. And how everything else is gonna be a, a derivative of that. And then on top of that, we just got the agent breakthrough, you know, with Open Claw, which is fantastic. Which is amazing and incredibly powerful.And then we just got the, the, um, the auto research, uh, you know, the, the self-improvement. You know, we’re now into the self-improvement breakthrough. And so the, so the way I think about it is we’ve had four fundamental breakthroughs in functionality, l OMS reasoning, uh, agents, um, and then, uh, and, and then now RSI, um, and, and they’re all actually working.Um, and so I’m, I’m just, as you like, you can tell I’m jumping outta my shoes. Like, like this is, like this is it like this, this is the culmination of 80 years worth of worth of work, and this is the time it’s becoming real.Alessio: Yeah.Marc: I, I’m completely convinced.Alessio: I think the anxiety that people feel is like during the transistor era, yet Mors law, and it’s like, all right, we understand why these things are getting better.We understand the physics of it. Yeah. With ai, it’s. It’s so jagged in like the jumps where like, like you said, it’s like in three months you have like this huge jump like, and people are like, well this can keep happening. Right? But then it keeps happening,Marc: it’ll keep happening.Alessio: And so like how do you think about also timelines of like what’s we’re building?I think we always have this question with guests, which is like, you know, should you spend time building harness for a model versus like the next model just gonna do it one shot in the lead space. Right. And how does that inform, like how you think about the shape of the technology? You know, you talk about how it’s a new computing platform.If you have a computing platform, then like every six months it like drastically changes in what it looks like. It’s hard to build companies on top of it.Marc: Yeah. So, so a couple things. So one is like, look, the, the Moore’s law was what we now call a scaling law. Like Moore’s Law was a scaling law and for your younger viewers, more Moore’s Law was every chip chip chips either get twice as powerful or twice as cheap every, every 18 months.And that, and that and that, you know, that it’s gotten more complicated in the last few years. But like that, that was like the 50 year trajectory of, of, of the computer industry. And then, and then by the way, and that’s what took the mainframe computer from a $25 million current dollar thing into, you know, the phone in your pocket being, you know, a million times more powerful than that.Like that, you know, for, for 500 bucks. And so that, that was a scaling law. And then, and then, and then key to any scaling law, including Moore’s Law and the AI scaling laws is, you know, they’re not really laws, right? They’re, they’re, they’re, they’re predictions, but when they work, they become self-fulfilling predictions because they, they, they, they, they set a benchmark and, and then the entire industry, right?All the smart people in the industry kind of work to make sure that, that, that actually happens. And so they, they kind of motivate the breakthroughs that are required to, to keep that going. And, and in and in chips, that was a 50 year, that was a 50 year run. Right. And it, it was amazing. And it’s still happening in, in some areas of, of chips.I think the same thing is happening with the, the core scaling laws. The core scaling laws. In, in, in ai, you know, they’re, they’re not really laws, but like they, they are basically. There are predictions and then they’re motivating catalysts for the research work that is required to be. And, and, and, and by the way, also the investment, uh, dollars, um, uh, you know, required to basically keep, you know, keep the curves going and, and look, it, it is, it’s gonna be complicated and it’s gonna be variable and they’re, you know, there’re gonna be walls that are gonna look like they’re fast approaching, and then they’re gonna be, you know, engineers are gonna get to work and they’re gonna figure out a way to punch through the walls.And obviously that’s, you know, that’s been happening a lot, you know, and then look, there’s gonna be times when it looks like the walls have, you know, the, the, the laws have petered out and then they’re gonna, they’re gonna pick up again and surge and then, and then, and then it, it appears what’s happening to the eyes is there’s not multiple, you know, multiple scaling laws.Um, there’s multiple areas of improvement. And, and I think, you know, I don’t know how many more there are already yet to be discovered, but there are probably some more that we don’t know about yet. You know, they, like, for example, there’s probably some scaling law around, um, world models and robotics that we don’t fully understand, you know, kind of acquisition of data at scale in the real world that we don’t fully understand yet.So that, that, that one will probably kick in at some point here. There’s a bunch of really smart people working on that. Um, and so, yeah, I, I think the expectation is that, that, you know, the, the scaling laws generally are gonna continue. Yeah. The, the pace of improvement will continue to move really fast.Um. To your question on like what to build. So, uh, I’m a complete believer the scaling laws are gonna continue. I’m a complete believer the capabilities are gonna keep getting amazing, um, you know, leaps and bounds. Uh, the part where I kind of part ways a little bit with how, what I would describe as the AI purists, um, you know, which is, which I would characterize as like the people who are.In many ways, the smartest people in the field, but also the people who spend their entire life, like at a lab, um, and have, have, I would say, have very little experience in the outside world. Um, the, the, the nuance I would offer is the outside world of 8 billion people and institutions and governments and companies and economic systems and social systems is really complicated.Um, and, um, and doesn’t, you know, it it 8 billion people making collective decisions on planet Earth is not a simple process of like, just like you see this happening now. It’s like a bunch of AI CEOs have this thing, which is just like, well, there’s just this, they just all have this kind of thing when they talk in public where they’re just like, well, there’s these, these obvious set of things that so society to do.Alessio: Mm-hmm.Marc: And then they’re like, society’s not doing any of those things. Right. And it’s like, how can society not, you know, what, whatever their theory is, how can society not see x, y, Z? Mm-hmm. And the answer is, well, society is number one. There’s no single society, it’s like 8 billion people. And they like all have a voice, and they all have a vote, like at the end of the day of how they, they react to change.And then, you know, it just like, it’s just human reality is just really complicated and messy. Um, and, and, and so the specific answer to your question is like, as usual, it depends. Um, you know, it, it depends. Look, pe there’s no question people are gonna, like, there’s no question they’re gonna be companies.It’s already happening. There are companies that think that they’re building value on top of the models and then they’re just gonna get blissed by the, by the next model. There’s no question that’s happening. But I think there’s no question also that just the process of adaptation of any technology into the real and into the real messy world of humanity is, is just going to be messy and complicated.It’s, it’s not going to be simple and straightforward. It’s gonna be messy and complicated. And there are gonna be a lot of companies and a lot of products, um, uh, and in, in fact entire industries that are gonna get built to, to, to basically actually help all of this technology actually reach real people.Alessio: The amount of capital going into these companies, I mean, Dario talked about it on the Door Cash podcast and Door Cash was like, why don’t you just buy 10 x more GPUs? And he is like, because I’m gonna go bankrupt if the model doesn’t exactly hit the, the performance level. How do you think about that?Also as a risk on, you know, you guys are investors, open AI and thinking machines and world apps. It seems like we’re leveraging the scaling loss at a pretty high rate, right? Like how comfortable, I guess, do you feel with the downside scenario, like, and say like things Peter out, you think you can kind of like restructure uh, these build outs and uh, you know, capital investments.Marc: Yeah. So should start by saying, so I live through the.com crash, um, and I can tell you stories for hours about the.com crash and it was horrible. No, it was awful. It was, it was, it was apocalyptic by the way. The, a lot of the.com crash was actually at the time, it was actually a telecom crash. It was a bandwidth crash.Like the, the thing that actually crashed, that wiped out all the money with the tele, the telecom companies.swyx: GlobalMarc: crossing. Global, global, yeah.swyx: I’m from Singapore and they, they laid so much cable o over over our oceans.Marc: Actually there was a scaling law in the.com. Era. And it was literally the, the US Commerce Department put out a report in 1996 and they said internet traffic was doubling every quarter.Um, and, and actually in 1995 and 1996, internet traffic actually did double every quarter. And so that became the scaling law. And so what all these telecom entrepreneurs did was they went out and they raised money to build fiber, anticipating that the demand for bandwidth is gonna keep doubling every quarter.Doubling every quarter though is like, you know, grains of chess and the chessboard, like at some point the numbers become extremely large. Right. And, and, and it really, and really what happened was the internet. The internet by the way, continuously kept growing basically since inception. And it’s, you know, it’s, it’s continuously grown.It’s never shrunk. And it’s grown really fast compared to anything else. Mm-hmm. You know, in, in, in human history. But it wasn’t doubling every quarter as of 19 98, 19 99. And so there was this gap in the expectation of what they thought was a scaling law versus reality. And that’s actually what caused the.com crash, which was the, it they, they way over companies like global crossing way overbuilt fiber, which is sort of the, and by the way, fiber, telecom equipment, you know, so all the, all the networking gear, you know, and then, and then by the way, the actual physical data centers, like that was the beginning of the, of the, of the data center build and then, and the data center overbuild.And so you had that, but it was, it was literally, I think it was like $2 trillion got wiped out, right? It was like Jesus, it was like a big, it was. And by the way, the other, the other subtlety in it was the internet companies themselves never really had any debt. ‘cause tech, tech companies generally don’t run on debt, but the telecom companies run on debt.Physical infrastructure companies run on debt. And so the companies like Global Crossing not just raise a lot of equity, they also raise a lot of debt. So they’re highly levered. And so then you just do the thing. It’s just like, okay, you have a highly levered thing where you’re, you’re just over, you’re overbuilding capacity.Demand is growing, but not as fast as you hoped. And then boom, bankrupt. Right. And, and then it, and then it’s like they say about the hotel industry, which is, it’s always the third owner of a hotel that makes money. It has to go bankrupt twice, right? You have to wash out all of the over optimistic exuberance before it gets to actually a stable state.And then it makes money. So by the way, all of those data centers and all of those, all the fiber that they’re in use, it’s all in use today. Yeah. But 25 years later. But it, it, it took, and actually the elapsed time was, it took 15 years. It took 15 years from 2000 to 2015 to actually fill, fill up all that capacity.The cautionary warning is the, the overbuild can happen. Um, and, and, and, and, you know, you, you get into this thing where basically everybody, everybody who basically has any sort of institutional capital, it’s like, wow. It’s just, I, I don’t know how to invest in these crazy software things. For sure I can put build data centers and for sure I can buy GPUs that I can deploy, you know, compute grids and, and all these things.Um, and so, you know, if you’re a pessimist, you could look at this and you could say, wow, this is like really set up to be able to basically replicate, you know, what we went through, what we went through in 2000. Obviously that would be bad. The counter argument, which is the one I I agree with, which is the counter on, on the other side is a couple things.One is the companies that are investing all the, the companies that are investing the money are like the bluest chip of companies. And so back, back, back in the, in the do, like Global Crossing was like a, it was like an entrepreneur. It was like a, a new venture, but like the money that’s being deployed now at scale is Microsoft, and, you know, and Amazon and Google, Facebook and Facebook and Nvidia and, you know, these, these, these, and, and now you know, by the way, open ai philanthropic, which are now at like, you know, really serious size, um, you know, as companies with, you know, very serious revenue.These are very large scale companies with like, lots, lots of cash, lots of debt capacity that they’ve, they’ve never used. And so th this is institutional in a way that, that really wasn’t at the time. And then the other is, at least for now, every dollar that’s being put into anything that results in a running GPU is being turned into revenue right away.Like so, and you guys know this, like everybody’s starved for capacity, everybody’s starved for compute capacity and then, you know, all the associated things, memory and, and, and interconnected and everything else. Um, data center space. And so e every dollar right now that’s being put into the ground is turning into revenue.And, and it, and in fact, I actually think there’s an interesting thing happening, which is because everybody starve for capacity, the models that we actually have that we can use today are inferior versions of what we would have if not for the supply constraints. That’s true. Um, if Right pose a hypothetical universe in which GPUs were 10 times cheaper and 10 times more plentiful mm-hmm.The models would be much better. ‘cause you would just allocate a lot more money to training and you’d just build better models and they would be better. Um, and so we’re, we’re actually getting the sandbag version of the technology.swyx: Yeah. No. Everything we use is quantized because the, the labs have to keep the, the full versions,Marc: right?swyx: LikeMarc: we’re not even getting the good stuff.swyx: Yeah.Marc: But, but getting the good stuff, it’s, it’s just, even if technical progress stops. Once there’s like a much bigger build of like GPU manufacturing capacity and memory, you know, all, all the things that have to happen in the course of the next five or 10 years.Once it happens, even the current technology is gonna get, gonna get much better. And then as you know, like there’s just like a million ways to use this stuff. Like there’s just like a million use cases for this. Mm-hmm. Like, it, it, you know, this isn’t just sending packets across a, a thing, whatever, and hoping that people find something to do with it.This is just like, oh, we apply intelligence into every domain of human activity. And then it works like incredibly well. Yeah. Um. Here’s what I know, here’s what I know. Um, in the next three or four year, it’s like somewhere between three or four years out, basically everything is selling out. So like the, the entire supply chain is, is, is, is sold out or, or, or selling out.And so there, there’s no, like, we’re just gonna have like chronic supply shortage for, you know, for years to come. Um, there’s going to be a response from the market that’s gonna result in an enormous, you know, it’s happening now. An enormous flood of investment in a new fab capacity and ev you know, every, everything else to be able to do that, at some point the supply chain constraints will unlock, you know, at least to some degree that will be another accelerant to industry growth when that happens.‘cause the products will get better and everything will get cheaper. Um, and so, so I know that’s gonna happen. I know that, you know, the deployments, you know, the, the actual use cases are like really compelling. And then, like I said, you know, with reasoning and agents and so forth, like, I know they’re just gonna get like much, much better from here.And so I, I, I know the capabilities are like really real and serious. I also know that the technical progress is not going to stop. It. It, it is excel. It is, is accelerating. Like the, the breakthroughs are are tremendous. I mean, even just month over month, the breakthroughs are really dramatic. And so, you know, I think if you were a cynic and there, there are cynics, you can look at 2000, you can find echoes.But I can’t even imagine betting it that this is gonna like somehow disappoint and, you know, at least for years to come, I think it would be essentially suicidal to make that bet. Yeah. Um, it was that Michael Burry, uh, uh, that’sswyx: anMarc: interesting guy, huh? We’ll pick on a guy. We’ll pick, let’s pick on one guy.We’ll pick. Well ‘cause he did, he he came out with, it was, it was the, heswyx: doesn’t mind.Marc: It was the Nvidia short. Right. He came with the Nvidia short. And then if you guys probably talked about this, which is the, the analysis now that like the current models are getting better faster at such a rate that if you are running an Nvidia, if you’re running an Nvidia inference chip today, that’s three years old, you’re making more money on it today than you did three years ago because the pace of improvement of the software is, is faster than the, the, the depreciation cycle, the chip.And then my understanding is Google is running. I don’t if they’ve, I don’t know exactly what, uh, these are rumors that I’ve heard or maybe it’s public, but, um, I think Google’s running very old TPUs, very profitably. Ference. Yeah. And very profit and very profitably. Yeah. Um, and so, so it actually turns out, as far as I can tell, it’s actually the opposite of the Beery thesis is actually.He was actually 180 degrees wrong. It’s actually the, the, the, the old Nvidia chips are getting more valuable, which is something that’s like literally never happened before. Like it’s never been the case that you have an older model chip that becomes more valuable, not less valuable. And that, and again, that’s an expression of the just ferocious pace of software progress.Ferocious pace of capability payoff. Yeah. Uh, that you’re getting on the other side of this. And so I just, the idea of betting against that, like.swyx: Yeah. Yeah. Well, one ofMarc: my, it seems like an invitation to get your face ripped up.swyx: One of my early hits was like modeling the lifespan of the H 100 and h two hundreds and, and going like, you know, usually they advise like four to seven years and it was, you know, maybe you sort of realistically haircut cut it down to two to three.Yeah. But actually it’s going up and not down. Yeah. And, and uh, that’s, I mean that’s, I think that’s the dream. Uh, we are finding utilization and I think utilization solves all problems. Like, you can, you can find use, use cases for even like the poor, like even memory, we’re having a shortage. Right. And, and even like the, the shittier versions of, of memory that we do have, we are finding use cases for it.So like That’s great.Marc: Yeah.Alessio: How, how important is open source AI and kinda like edge inference in a world in which you have three years of supply crunch. Like, do you think in the, like, you know, if you fast forward like five years, like how do you think about inference, uh, in the data center versus at the edge?Marc: Well, so just to start, yeah. So I think, I think open source is very important for a bunch of reasons. I think edge, edge inference is very important for a bunch of reasons. I, I think just practically speaking, if we’re just gonna have fundamental construc, supply crunches for the next, I mean, you, you guys know if you just project forward demand over the next three years, right?Yeah. Relative to supply, one of the, its main predictions you can do is what’s gonna, what, what’s gonna happen to the cost of, of inference in the core, uh, over the next three years? And like, it may rise dramatically, right? Like, so, so what is, and then is, is, you know, like the, the, the big model competition are subsidizing heavily right now.Right? Right. And so, so what’s the, what will be the average person’s, you know, per day, per month token cost, you know, three years from now to do all the things that they want to do. And I, I don’t know, it’s gonna. I mean, I have, you guys probably have friends, I have friends today who are paying a thousand dollars a day for open claw, for claw tokens to run open claw.Right? And so, okay. $30,000 a month. Right? And, and by the way, those, those friends have like a thousand more ideas of the things that they want their claw to do, right? Yeah. And so you, you could imagine there, there’s like latent demand of up to, I don’t know, five or $10,000 a day of, of, of tokens for a fully deployed, you know, per personal agent.Uh, and obviously consumers can’t pay that, right? And so, so, but it gives you a sense of the fu of the fu of the future scope of demand, right? And so, so even, even if there’s a 10 x improvement in price performance, that still, you know, goes to a hundred dollars a day, which is still way beyond what people can pay.Mm-hmm. So there’s just gonna be like. Ferocious to me, by the way. The agent thing, the other interesting thing is I think the agent thing, so up until now, a lot of the constraints of GGPU constraints, I think the agent thing now also translates into CPU constraints. Mm-hmm. Right?swyx: CPU memory.Marc: Yes. CPU memory, right?And so, like the entire chip ecosystem is just gonna get wait,swyx: wait for network constraints, that that will be the killer.Marc: It’s all bottleneck potentially for years. And so, so I, I think that Brad, and, and I think it’s actually possible, I mean, generally inference costs are gonna keep coming down, but I think the, let’s put it this way, the rate of decline, I think may level out here for a bit because of these supply constraints.And then at some point, maybe the lab stops subsidizing so much and that, that, that again, will be, be an issue. And so there’s just gonna be so much more demand for inference than, than can be satisfied. Um, you know, kind of with the centralized model. And then, and then, you know, you guys know this, but like all the, just the dramatic, I mean just the dramatic innovations that have happened in the Apple silicon to be able to do, uh, inferences, it’s quite amazing the level of effort being put.Like the open source guys are putting incredible effort into getting, you know, this recurring pattern where the big model will never run on a pc, and then six months later mm-hmm. Oh, it runs in a pc, right? It’s like amazing. And there’s very smart people working on that. So there’s all that. And then look, there’s also, you know.There’s also like other, there’s other motivators. There’s other motivators which is just like, okay, how much trust are the big centralized model providers? You know, how much trust are they building in the market versus, you know, how much are, you know, at least for, in certain cases with some people, for certain use cases, people being like, well, I’m not willing to just like, turn everything over.So there, there, there’s all the trust issues. Um, by the way, there’s also just like straight up price optimization. There’s many uses of AI where you don’t need Einstein in the cloud. You just need like a, a a, a smart local model. There’s also performance issues where you want, you know, you want, you know, you’re gonna want your doorknob to have an AI model in it.Right. You know, to be able to, you know, do, um, you know, to be able to do access control. Um, obviously like everything with a chip is gonna have an AI model in it. Mm-hmm. And it, a lot of those are gonna be local. Um, and so, yeah. No, like I think, I think you’re gonna have ti and then you’re gonna, by the way, also wearable devices, you know, you don’t wanna do a complete round trip.You want, you know, you, whatever your smart devices are, you want it to be like super low latency. Yeah.swyx: The question, do we care who makes it? Yeah. One of the biggest news this week was the collapse of AI two, the Allen Institute. Mm-hmm. One of the actual American open source model labs. Yeah. Um, and, uh, I’m not that optimistic on, on American open source.Yeah. Like you, you guys invested in MIS trial and MIS trial’s doing extremely well outside of China. That’s about it.Marc: Yeah. We’ll see. We’ll see. I look, I, number one, I do think we care. Uh, I do think we, I do think we care who makes it. Um, I would say this, the, the, the, the previous presidential administration wanted to kill it in the us Oh yeah.They wanted to drown in the bathtub. Um, and so they wanted to kill it. So at least we have a government now that actually like, actually wants it wants it to happen. And youswyx: earned to councilMarc: and Yeah. And the new and the P pcast. Yeah. So the, the, you know, this admin for whatever other political issues people have, which are many, you know, this administration has, I think a very enlightened view and in particular an enlightened view on AI and in particular on open source ai.Uh, and so they’re very supportive. Um, my read is the Chi. The Chinese have a very, the various Chinese companies have a very specific reason to do open source, which is, they, they, they don’t fundamentally, they don’t think they can sell commercial, uh, AI outside of China right now. And or at least specifically not, not in the US for a combination of reasons.And so they, they kind of view, I think, open source AI as a bit of a loss leader against basically domestic, uh, you know, paid, paid services. And then kind of an, you know, kind of an ancillary products. You know, they’re, they’re very excited about it, by the way. I think it’s great. I think it’s great that they’re doing it.Um, you know, I think Deeps seek was like a gift to the world. Um, I think. The great thing about open source, open source, the, the, the impact of open source is felt two ways. One is you, you get the software for free, but the other is you get to learn how it works, right? And so like the paper, the paper, the paper and, and the code, right?And the code. And so, like, for example, I thought this was amazing. So open comes out with L one and it’s an amazing technical breakthrough, and it’s just like, absolutely fantastic. But of course they don’t explain how it works in detail. And then of course they hide the, they hide the reasoning traces, right?And, and then, and then, and then everybody’s like, okay, this is great, but like, who’s gonna be able to replicate this? Are other people gonna be able to do this? You know, is their secret sauce in there? And then our one comes out and it’s just like, there’s the code and there’s the paper, and now the whole world knows how to do it.And then, you know, three months later, every other AI model is, is adding reasoning. And so, so you get this kind of double, like even if the Chinese models themselves are not the models that get used, the education that’s taken place to the rest of the world, the information diffusion, you know, is incredibly powerful.So that happens and then, I don’t know. We’ll, we’ll see. You know, there are a bunch of American, you know, open source, you know, ai, uh, model companies. I mean, look, there’s gonna be tremendous, you know, there already is. There’s, you know, there’s gonna be tre there’s tremendous competition, uh, among the primary model companies.You know, there’s, depending on how you count, there’s like four or five, you know, big co model companies now that are, you know, kind of neck and neck, uh, in different ways. Um, uh, you know, and, and, and, um, you know, and then obviously Bo Bo both X and then MetAware involved are, you know, both have huge, you know, huge attempts to, you know, kind of, to kind of leapfrog underway.And then you’ve got, you know, a whole fleet of startups, new companies, including a whole bunch that we’re backing, that are, you know, trying to come out with different approaches. And then you’ve got whatever it is. I don’t know how, how many, how many, like main line foundation model companies are there in China at this point?It’s probably six. It’sswyx: five Tigers is what they call it. Yeah. Uh, Quinn is in questionable because there’s change in leadership,Marc: right?swyx: Yeah.Marc: But that, does that include, that includes like Moonshot,swyx: yes. Can deep seek, uh, uh, ZI, um, Quinn oh one is in there.Marc: Right. And then, um, and by dance and, and then you see,swyx: ance would be like the next tier ance.They weren’t as prominent. They weren’t, didn’t haveMarc: a leading. Yeah. But they, you at least, you know, ance is very inspiring and presumably they have more stuff coming and Tencent probably has more stuff coming and, and so forth. And so, so, so like, look, here, here would be a thing you can anticipate, which is there are not these markets, there are not going to be between the US and China right now, there’s like a dozen primary foundation model companies that are like at scale, at, at some level of a critical mass.It’s not gonna be a dozen in three years, right? Like, it just because these industries don’t bear a dozen, it’s, it’s gonna be three or you know, there’s gonna be three or four big winners or maybe one or two big winners. And so there’s gonna be like a whole bunch of those guys that are gonna have to figure out alternate strategies.Um, and I think like open source is one of those strategies. And so I, I think you could see like a whole, i, I, I think the questions like, who’s gonna do open source? I think that could change really fast. I, I think that, that, that’s a very dynamic thing. I think it’s very hard to predict what happens. And, and I think it’s very important.swyx: NVIDIA’s doing a lot.Marc: Well, I was gonna say. Well, exactly. And then you’re got Nvidia and then, and then, you know, just to, again, indu, there’s an old thing in business strategy, which is called, uh, commoditize Compliments. Commoditize the compliment. That’s right. And so if your Jensen is just kind of obvious, of course, you wanna commoditize the software.Yeah. And he’s, and to his enormous credit, he’s putting enormous resources behind that. And so maybe it, maybe it’s literally Nvidia and I think that would be great.Alessio: Yeah. Uh, narrative violation to European projects, uh, in the, uh, damn.swyx: I’m hosting my, uh, Europe, uh, conference soon. And I got both of them.Alessio: They got us.They got us. MarkMarc: finished. They got us, us. Well, wait a minute. Where was Peter? So where was Steinberger when he did? In AustriaAlessio: was, yeah, yeah, yeah.Marc: He was in what? He was in Vienna. Oh, he was in Vienna. And then where is he now?swyx: Uh, he’s moving to sf.Marc: Okay. Okay. Alright. Okay, there we go. And then, yeah, the PI guy, right?The PI guys are European.swyx: Yeah, they’re also, they’re buddies inAlessio: Australia. Mario’s also there. Yeah.Marc: Right. And are they, yeah, they haven’t announced yet. Any sort of change changed or have theyAlessio: No, they’re, they have a company there.Marc: Okay. Got, okay. Good.Alessio: Good, good,good.Alessio: Um,Marc: yeah, good.swyx: Anyways, I think pie and open cloud very important software things and, and I just wanted you to just go off on what you think.Marc: Yeah. So I think in co the, the combination of the two of them I think is one of the 10 most important softwares. Openswyx: Claw got all the attention, but Right. Talk about pie,Marc: pi pie’s, kind of the Yeah. PI’s, PI’s kind of the architectural breakthrough for those of us who are older. There was this whole thing that was very important in the world of software basically from like 1970 to, I don’t know, it still is very important, but like 19, from 1973 to like basically the creation of Linux, which is basically this, this thing used to call like the Unix mindset.Like so, so, ‘cause there were all these different, you know, theories. There are all these different operating systems and mainframes and, and then you know, all these windows and Mac and all these things. And then there was this, but kind of behind it all was this idea of kind of the Unix mindset. And the Unix mindset was this thing where basically you don’t have these, like, like in the old days, like, like the operating system that like made the computer industry really work, like in the 1960s mm-hmm.Was this thing called o os 360, which was this big operating system that IBM developed that was supposed to basically run everything. And it was this like giant monolithic architecture in the sky. It was like a, you know, it was like a giant castle. Um, of software. And, and by the way, it worked really well and they were very successful with it.But like, it was this huge castle in the sky, but it was this thing, it was almost unapproachable, which is like, you had to be kind of inside IBM or very close to IBM. And you had to really understand every aspect, how the system worked. And then the, the Unix sky is originally out of at and t and then out out of Berkeley, um, you know, came out and they said, no, let’s have a completely different architecture.And the way architecture’s gonna work is we’re gonna have, we’re gonna have a, a prompt and, and a, and a shell. And then, and then we’re gonna, all, all the functionality is gonna be in the form of these discreet modules, and then you’re gonna be able to chain the modules together. Mm-hmm. Yeah. And so like the, the, the op, it’s almost like the operating, operating system itself is gonna be a programming language.Um, and then that led led to the, the, the sort of centrality of the shell. Um, and then that led to sort of, uh, you know, basically chaining together Unix tools. And then that led to the emergence of these, these scripting languages like Pearl, where you, you could basically kind of very easily do this, and then the shells got more sophisticated and then, and then, and then look like, you know, that, that, that number one, that worked and that, that was the world I grew up in.Like I was, I was a Unix guy. You know, sort of from, call it 1988 to, you know, kind of all, all the way through my work and it worked really well. It, it’s in the background, um, you know, nor normal people don’t need to, didn’t need to necessarily know about it, but like, if you were doing like system architecture, application development, you, you, you knew all about it.Um, and then, you know, it’s been in the background ever since. And, you know, look, your Mac still has a Unix shell, you know, kind of in there, and your iPhone still has a Unix shell kind of buried in there somewhere. So they’re kind of in there. And then, you know, the Windows shell is kind of a, you know, sort of a weird derivative of that.But, um, you know, but look, the inter, the internet runs on Unix, um, and that smartphones, actually, both iOS and Android are Unix derivatives. And so, you know, kind of Unix did end up winning. But, but anyway, and then we just started taking that for granted. And then, and then so, so basically the, the way I think about what happened with Pie and then with Open Claw is basically what those guys figured out is, I always say the, the great breakthroughs are obvious in retrospect, right?Which is the best kind, the best kind. They weren’t obvious at the time or somebody else would’ve done them already. Um, and so there is a, like a real conceptual leap, but then you look at it sort of the backwards looking and you’re just like, oh, of course. Mm-hmm. Like the, the, to me those are always the best breakthroughs.Well, actually language models themselves are like that. It’s just like, oh, next token completion. Oh, of course.swyx: Yeah. What other objective mattered?Marc: Yeah, exactly. But, but like it, right. But she’s even saying it wasn’t obvious until somebody actually did it. Right. And so the conceptual breakthrough is real and deep and powerful and, and very important.And so the way I think about pie and olaw is it’s basically marrying the, the language model mindset to the un to the Unix, basically shell prompt mindset. And so it’s, it’s basically this idea that what, what, so what is an agent, right? And as, as, and as you know, like many smart people who have been trying to figure out what an agent is for, for, for decades, and they’ve had many architectures to build agents and the whole thing.And it turns out what is an agent. So it turns out what we now know is an agent is the following. It’s, so it’s a language model. And then above that, it’s a ba, it’s a bash shell. Um, so it’s a, it’s a Unix shell, and then it’s, and then the agent has access, uh, has access to, to the shell. And, you know, hopeful, hopefully in a sandbox, maybe in, maybe in a sandbox.So it’s, it’s the model. Um, it’s the shell. Um, and then it’s a fi, it’s a file system. Um, and then the state is stored in files. And then, you know, there’s the markdown format for the, you know, for, for the files themselves. And then, and then there’s basically what in Unix is called Aron job. There’s a loop and then there’s a heartbeat for the, there’s heartbeat and, and the thing basically Wake Wakes up.Wakes up. So it’s basically LLM plus shell, plus file system, plus markdown, plus kron. And it turns out that’s an agent. And, and, and every part of that, other than the model is something that we already completely know and understand. And in fact, it turns out that like the latent power of the Unix shell is like extraordinary because basically like all, like, there’s just like an, there’s just enormous latent power in the shell.There’s enormous numbers of Unix commands, there’s enormous number of command line interfaces into all kinds of things already in the, you know, your entire, I mean your entire, just to start with, your computer runs on a shell. If you’re running a Mac or a, or, or a phone, your computer, your computer’s running on a shell, uh, already.And so like the full power of your computer is available at the command line level. Um, and then it turns out it’s really easy to expose other functions as a command line interface. And so like this whole idea where we need like MCP and these like product mm-hmm. Fancy protocols, whatever, it’s like, no, we don’t, we just need like a command, command line thing.So that’s the architecture. And then it turns out what is your agent? Your agent has a bunch of files starting a file system. And then there’s the thing that just like completely blew my mind when I write my head around it as a result of this, which is like, okay. This means your agent is now actually independent of the model that it’s running on.Because you can actually swap out a different LLM underneath your agent and your, your agent will change personality somewhat. ‘cause the model is different, but all of the state stored in the files will be retained.swyx: Yeah. Different instruction set, but you just compiledit.Marc: Right, exactly. And it’s all right.It’s like right. Swapping out a ship and recompiling, but it’s, it’s still, it’s still your agent with all of its memories. Um, and with all of its capabilities. And then by the way, you can also swap out the shell, uh, so you can move it to a different execution environment that is also, is also a b shell, by the way, you can also switch out the file system, right.Uh, and you can, and you can, and you can swap out the, the, the heartbeat for the, the crown framework, the, the loop that the agent framework itself. And so your agent basically is ba basically at the end of the day, it’s just. It’s just, its files. Um, and then, and then there’s of course it a openswyx: call.Marc: Yeah, it’s, it’s basically, it’s, it’s just the files.Um, and then by the way, as a consequence of that, the agent and then the agent itself, it turns out a couple important things. So one is it, it’s, it, it can migrate itself, right? And so you’re, you can instruct your agent, migrate yourself to a different, uh, runtime environment, migrate yourself to a different file system, migrate yourself to a different, you know, swap out the language model.Your agent will do all that stuff for you. And then there’s the final thing, which is just amazing, which is the agent is the agent actually has full introspection. It actually, it actually knows about its own files and it could rewrite its own files. Right. Which by the way, is basically no widely deployed software system in history where the, the, the thing that you’re using actually has full introspective knowledge of how it itself works and is able to modify itself.Like that, that, I mean, there have been toy systems that have had that, but there, there’s never been a widely deployed system that has that capability and then that leads you to the capability. That just like completely blew my mind when I wrap my head around it, which is you can tell the agent to add new functions and features to itself and it can do that.Extend yourself. Yeah. Right? Extend, extend yourself. Like extend yourself. Give yourself a new capability. Right? And so, and so literally it’s just like you run into somebody at a party and they’re like, oh, I have my open claw, do whatever, connect to my eat, sleep bed, and it gives me better advice and sleep.And you go home at night and you tell your claw, or if they’re at the party, by the way, you tell your claw, oh, add this capability to yourself. And your claw will say, oh, okay, no problem. And it’ll go out on the internet and it’ll figure out whatever it needs and then it’ll go out to claw code or whatever.It’ll write whatever it needs. And then the next thing you know, it has this new capability. And so you don’t even have to, like, you can have it upgrade itself without even having to, without having to do anything other than tell it that you want it to do that. And so anyway, so the, the combination of all this is just, I mean, this is just like a massive, incredible, I mean, it’s just incredible.Like if I, if I were, if I were 18, like this is a hundred, this is what I would be spending all of my time on. This is like such an incredible conceptual breakthrough. Yeah. And again, pe people are gonna look at it and they already get this response. People are gonna look at it and they’re gonna say, oh, well, where’s the breakthrough?‘cause these, the, all of these components were already known before. Mm-hmm. But, but this is the key, the key to the breakthrough was by using all these components that were known before, you get all of the underlying capability of that’s buried in there. And so all, and so for example, computer use all of a sudden just kind of falls, trivi, trivial.Of course it’s gonna be able to use your computer. It has full access to the shell. Right. And then, and then you just, you, you give it access to a browser, and then you’ve got the computer and the browser and, and often away it goes. And, and then you’ve got all the abilities of the browser also. Um, yeah.And so, and so the capability unlock here is profound. My friends who are, you know, deepest into this, are having their claw do like a, like, literally like a thousand things in their lives. They have new ideas every day. They’re just like constantly throwing new challenges at the thing. And by the way, it’s early and, you know, these are, you know, these are prototypes and there are, you know, as you guys know, there’s security issues.Yeah. And, and so, you know, there’s a bunch of stuff to be ironed out, but the, the unlock of capability is just incredible.swyx: Yeah.Marc: And I, I have absolutely no doubt that everybody in the world is gonna, is gonna have at least, you know, an agent like this, if not an entire family of agents. And we’re gonna be living in a world where I think it’s almost inevitable now that this is the way people are gonna use computers.swyx: I was gonna say for someone who is deeply familiar with social networks, the next step is your claw talking to my Claw. Mm-hmm.Marc: Postingswyx: on Claw Facebook, uh, posting their jobs on cloud LinkedIn and close posting their tweets on claw XAI or what, whatever, you know. Um, I do think that that is how, uh, you know, we, we get into some danger there in, in terms of like alignment and whether or not we want these things to, to, to run.Marc: You guys know where Rent a, rent a human.com.swyx: Yeah. Rent a,Marc: yeah. Yeah.swyx: I mean, it’s Fiverr, it’s TaskRabbit.Marc: Sure, of course.swyx: MechanicalAlessio: Turk.Marc: Yeah. But flipped, right. The agent hiring the people.Alessio: Yeah.Marc: Which of course is gonna happen, right? It’s obviously gonna happen.Alessio: I’m curious if you have any thoughts on the engineering side.So when you build the browser, the internet, you know, just a bunch of mostly plain text file plus some images, and today the, every website and app is like, so complex. Somehow, you know, the browser kept evolving to fit that in. Mm-hmm. Are there any design choices that were made like early in the browser and kinda like the internet and the protocols that you’re seeing agents similar to this?Like, Hey, this thing is just not gonna work for like this type of new compute and we should just. Rip it out right now.Marc: There were a whole bunch, but I’ll give you a couple. So one is, um, and we didn’t, you know, to be clear like this, this was not, you know, this is totally different. We didn’t have the capabilities we have today, but because Wet have, we didn’t have the language models underneath this, but, um, we did have this idea that human readability actually mattered a great deal.Um, and, and, and so, and specifically in those days, it was, it was not so much English language, but it was there, there was a design decision to be made between binary protocols and text protocols. And basically every, every, every basically old school systems architect that had grown up between like the 1960s and the 1990s basically said, you know, the internet, it’s, what do you know about the internet?It’s star for bandwidth. You, you just, you have these very narrow straws. Uh, you know, look, people, when we did the work on Mosaic, like pe, people who had the internet at home had a 14 kilobit modem, right? So you’re, you’re trying to like hyper optimize every bit of data mm-hmm. That, that travels over the network.And so obviously if you’re gonna design a protocol like HGTP, you’re gonna want it to be binary, you know, highly compressed, binary protocol for maximum efficiency. And you’re gonna wanna have it be like a single connection that persists. And you’re, you’re, the last thing you’re gonna wanna do is like, bring up and tear down new connections.And you definitely, you’re not gonna, not gonna want a text protocol. And so of course we said no. We actually want to go completely the other direction. It’s obviously, we only want text protocols. Uh, by the way, same thing in H TM L itself. We want html to be relatively verbose. You know, we want the tags to actually be like human readable.Um, we wanna useswyx: the most inefficient things possible.Marc: Yeah, we wanna do the, we wanna do the in, we wanna do the inefficient things.swyx: You’re the original token Mixer.Marc: Yeah, exactly. Yeah, yeah, yeah. Basically it’s just like better lessonAlessio: filled.Marc: Well, yeah. Well actually this was, this was actually the, the conscious thing, which basically says just like assume, assume a future of infinite, infinite bandwidth built for that, right?And then basically what it was, is it was a bet that it, it was a bet that if the system, if the, if the latent capabilities of the system were powerful enough, and that was obvious enough to people that would create the demand for the bandwidth that would cause the supply of bandwidth to get built that would actually make the whole thing work.And then specifically what we wanted was we wanted everything to be human readable because we, at the engineering level, we wanted people to be able to read the protocol coming over the wire and be able to understand it with their, with their bare eyes without having to like disassemble it or whatever.Right. Have it converted outta binary. Right. And so the, the, the, all the pro, you know, HTTP and everything else were, were, it was always, uh, text protocols. Uh, and the same thing with HTML and in, in many ways, some people say that the key breakthrough in the browser was the view source option, um, which is every webpage you go to, you could view source, which means you could see how it worked, which means you could teach yourself how to build right new, uh, to, to build new webpages.There was that. So human readability. Um, and, and again, human readability in those days still meant technical, you know, specs. You know, now it means English language, but there’s an incredible latent power in giving everybody who uses the system the option to be able to drop down and actually understand and see how it’s working.And that worked really well for the web and I think it’s working really well for ai. That was one. Um, what was the other, um. A big part of the idea of web servers was to actually surface the underlying latent capability of the operating system and to be able to surface the, uh, also the underlying latent capability of the database because basically what was a web server?What, what, what, what is a web server? Fundamentally? Architecturally, it’s, it’s, it’s the operating system. So it’s, it’s the operating system’s ability to, you know, it’s running on top of an os. So it’s the OSS ability to manage. The file system and do everything else that you wanna do, process everything. Um, and then of course, a lot of early, you know, a lot, a lot of websites are, are front ends to databases.Um, and so you wanted to, you wanted to unleash the underlying latent power of whether it was an Oracle database or some other, you know, some other Postgres or whatever, whatever it was. Um, and so a lot of the function of the web server was to just bridge from that internet connection coming in to be able to unlock the underlying power of the OS and the database.Uh, and again, people looked at it at the time and they were like, well, is this really, does this really matter? Like, is this important Because we’ve had databases forever and we’ve always had, you know, user interfaces for databases and this is just another user interface for a database. And it’s like, okay, yeah, fair enough.But on the other side of that is just like, this is now a much better interface to databases and one that 8 billion people are going to use and is going to be like, far easier to use and far more flexible. And, and, and, and you’re not just gonna have old databases. Now you have a system where people can actually understand why they want to build, you know, a million times more database apps than they have in the past.And then the number of databases in the world exploded. And so again, this goes to this thing of like building, building in layers. Some of the smartest people in the industry look at any new challenge and they’re like, okay, I’m, I’m, I need to build a new kind of application. So the first thing I need to do is build a new programming language, right?And then the next thing I need to do is build a new operating system, right? And then the next thing I need to do is I need to build a new chip. Right? And they, they kind of wanna reinvent everything. And I’ve, I’ve always had, maybe it’s just, I don’t know, pg pragmatic mentality or something, or maybe an engineering over science mentality, but it’s more like, no, you have just like all of this latent power, uh, in the existing systems and you, you don’t want to be held back by their constraints, but what you wanna do is you wanna kinda liberate that power and open it up.Yeah. And so I, I think, I think, and I think the web did that for those reasons. And I think it’s the same thing now that’s happening. It’s a greatswyx: perspective on the web.Alessio: Programming language just is not a good thing. We have Brett Taylor on the podcasts and we were talking about rust. And you know, rust is memory safe by the phone.So why are we teaching the model to not write memory, unsafe code, just use rust, and then you get it for free. How much do you think there’s like. Time to be spent like recreating some of these things instead of taking them for granted. I’ll be like, oh, okay. Python is kind of slow Pythonswyx: type scripts,Alessio: you know?It’s like, yeah.swyx: As, as imperfect as they are, they are the lingua franca.Marc: I mean, I think this is gonna change a lot. ‘cause I don’t think the models care what language they program in. Mm-hmm. And I think they’re gonna be good at programming in every language, and I think they’re gonna be good at translating from any language to any other language.Like, okay, so this gets into the coding side of things. I, I think we’re going through a really fundamental change. And then, look, I, I grew up hand, you know, I grew up hand code, you know? Yeah, yeah, yeah. I grew up hand coding. Everything I did was actually everything I did actually was written in CI wasn’t evenAlessio: back in the days,Marc: I wasn’t even using c plus plus, so I, or like Java or any of this stuff.Right. Uh, and so, um, I, everything, everything I ever did, I was like managing my own memory at, at, at the level of c and then I, you know, I, I’m still from the generation that, you know, I, I knew assembly language and, you know, I, I, you know, um, so I, I could drop down and do things, uh, right on the ship. And so we, we’ve just, we’ve all, all of us, we’ve always lived in a world in which software is like this precious thing that like, you have to think about very carefully.And it’s like really hard to generate good software. And there’s only a small number of people who can do it. And like, you have to be very, like, jealous in terms of thinking about like, how do you allocate, like what are your engineers working on and how many good engineers do you actually have? And how much software can they write?And how can, how much software can human beings, you know, kind of maintain? And I think like all those assumptions are being shot right out the window right now. Like, I think they’re, I, I think those days are just over. And I think the new world is like, actually high quality software is just like infinitely available.Mm-hmm.Marc: And if you need new software to do X, Y, Z, like, you’re just gonna wave your hand and you’re gonna get it. And then if it’s, if you don’t like the languages written in, you just tell the thing, all right, I want the, now I want the rush version. Um, or, you know, se secure, you know, secure. We’re about to, by the way, we’re about to go through computer security is about to go through the most dramatic change ever, which is number one, like every single latent security bug is about to be exposed,swyx: right?Marc: So we’re gonna have like, the in, we’re, we’re, we’re set up here for like the computer security apocalypse for a while. But, but, but on the other side of it, now we have a coding agents that can go in and actually fix all the security bugs. And so how, how are you gonna secure a software in the future?You’re gonna tell the, tell the bot to secure it, and it’s gonna go through and, and fix it all. And so, so this thing that was this incredibly scarce resource of high quality software is just going to become a completely fungible thing that you’re just gonna have as much as you want, right? Uh, and, and that has like, you know, that has like tons and tons of consequences in some sense.The answer to the question that you posed, I, I think it’s just somewhat, I don’t know, simple or something, or straightforward, which is just, if you want all your software and rust, you just, all the bot, you want all your software and rust, like, things that used to be like hard or even like, seem like an insurmountable mountain to get to get through all of a sudden, I think, become very easy.swyx: I, I think Brett had a theory that there would be a more optimal language for lms. And so the contention is, uh, there isn’t like, just don’t bother, just whatever humans already use LMS are perfectly capable, porting.Marc: I think we’re pretty close to being, I don’t know if this would work today. I think we’re pretty close to being able to ask the AI what would its opt optimal language be and let Right, and let it design it.True. Okay, here’s a question. Are you gonna even gonna have programming languages in the future? Um, or the ai, are the AI just gonna be emitting binaries? Let’s assume for a moment that humans aren’t coding anymore. Let’s assume it’s all bots. The bot. What levels of intermediate abstraction do the bots even need?swyx: Yeah.Marc: Or are they just coding binary directly? Did you see there’s actually an experi, somebody just did this thing where they have a, they have a, a language model now that actually emits model weights for a new language model. Right. And so will the bots be justAlessio: predict the weightsMarc: Will, yeah. Will the bots literally be emitting not just coding binaries, but will they, will, will they actually be admitting weights for, for new models?Yeah. Direct directly and. Conceptually, there’s no reason why they can’t do both of those things. Uh, like architecturally. Both of those things seem completely possible. It’sswyx: very inefficient. You’re basically veryMarc: inefficient.swyx: A simulation of a simulation in a simulation inside of the weights. Correct?Marc: Yeah, yeah. Very inefficient. But like, look, LMS are already like incredibly inefficient. Ask an uh, in favor thing, ask Claude, add two plus two equals four. Right? It’s just like, you know, it’s like, you know, it’s, it’s, it’s like whatever, billions and billions of times more inefficient than using your pocket calculator.swyx: Yeah.Marc: But, but, but yet the, the, the payoff is so great of the general capability. And so anyway, like I, I kind of think in 10 years, like, I’m not sure. Yeah. Like, I’m not sure there will even be a salient concept of a programming language, um, in the way that we understand it today. And in fact, what we may be doing more and more is a form of interpretability, which is we’re trying to understand why the bots have decided to, uh, structure, uh, code in the way that they have.swyx: I mean, if you play it through, you don’t need browsers, then like, that’s the depth of the browser.Marc: Well, so I, I would take it a step further, which is you may not need to use your interfaces. So who is gonna use software in the future?swyx: Other bots.Marc: Other bots. Yeah. Yeah. Andswyx: so you still need to, I don’t know, pipe information in,Marc: do we?swyx: And outMarc: reallyswyx: well, what are you gonna do then?Marc: Are you sureswyx: you’re just gonna log off and touch grass?Marc: Whatever you want. Exactly. Isn’t that better?swyx: I want software to do stuff for me.Marc: Isn’t that? But isn’t that better? I mean, look, I, you know, I don’t know. Look like, you know, you know, you, all the arguments here, you know, it was not that long ago that 99% of humanity was behind a plow.swyx: Right.Marc: Right. And what are people gonna do if they’re not plowing fields all day to, to, to grow food? Right. And it just turns out there’s like much better ways for people to spend time than plowing fields. Yeah.swyx: Dooms growing.Marc: Uh, yeah, exactly. Exactly. Or, you know, talking to their friends and look, and I’m not an absolutist and I’m not a utopian.And I, and to be clear, like I’ve, I have an 11-year-old and he’s learning how to code and like I’m, you know, I, I think it’s still a really good idea to learn how to code and so forth, but I just, if you project forward, you just have to think forward to a world in which it’s just like, okay, I’m just gonna tell the thing what I need and it’s gonna do it, and then, and then it’s gonna do it in whatever way is most optimal for it to do it.Mm-hmm. Yeah. Unless I tell it to do it non optimally. Like if I tell it to do it in Java or in Rust or whatever, it’ll do it, I’m sure. But like, if I’m just gonna tell it to do, it’s, gonna do it in whatever way is like the optimal way to do it. Yeah. And then I, and then if I need to understand how it works, I’m gonna ask it to explain to me how it works.Right. And so it’s gonna be doing its own, interpret it, it’s gonna be the engine of interpretability to explain itself. And I, I just am not convinced that, that I’m not, I’m not convinced that in that world you have these historical, the goals of the abstractions will be whatever, the Boston network with the human Right.Alessio: Yeah. Yeah. That, well, I, I’m curious like. If that’s true, then shouldn’t the models providers be building some internal language representation that they can do extreme, kinda like rl uh, and reward modeling around, because it’s like, today they’re kind of like tied to like type script and Python because the users need to write in that language versus they can have their own thing internally and like they don’t need to teach it to anybody.They just need to teach their model. And I think that’s how you get maybe the version between the models, like going back to like the pie open claw thing. It’s like, oh, I built all the software using the open AI model and now switch to the RO model. But the TRO model doesn’t understand the thing. So I I, it feels like there still needs to be some obstruction.But maybe not. Maybe that’s the lockin that the model providers want to have. I don’t,Marc: I’m not even sure that’s lockin though. ‘cause why can’t the second model just learn what the first model has done? Like,swyx: exactly.Marc: Okay. So okay. Give you an example. So as you know, models can now reverse engineer software by, right?Isn’t it the whole thing now where people are reverse engineering, like Nten, Nintendo, gay binaries. Yeah. So you, you have like there’s, I’ve seen a bunch of reports like this where somebody has like a favorite game from the 1980s and the source code is like long dead, but they have like a binary brand to do a chip or something, another reverse engineer to get a version that runs in their Mac.Right. And so if you reverse it, if, this is why I kinda say if you’re reversing like X 86 binaries, then why can’t you reverse engineerAlessio: whatever the degree. Yeah. And because we’re all on a Unix based system, it has to be reversible because it needs to run on the target.Marc: Yeah, yeah, yeah, yeah, yeah. Basically.And so I just, I just think it’s this thing where it’s just like, and by the way, and everything we’re describing is something that human beings in theory could have done before, but just with like, right. Yeah, yeah. But with enormous where, but it was just always like cost and labor prohibitive. Reverse engineer.I learned how to reverse engineer. Human beings can reverse engineer binaries. Yeah. It’s just for any complex binary, you need like a thousand years mm-hmm. To do it. But now with a model, you don’t. And so all of a sudden you get, you get these things. Or, or another way to think about it is so much of human built systems are to compensate for the human limitations.swyx: Mm-hmm.Marc: Yep. Right? Um, and if you don’t have the human limitations anymore, then all of a sudden you have, and, and it’s not that you, you won’t have abstractions, but you’ll have a different kind of abstraction. Yep. Yep.swyx: I have two topics to bring us to a close. And, uh, you could pick whichever ones. Uh, just talking about protocols, was it you or someone else?Uh, I forget my internet history. Who said that? Like the biggest mistake that we didn’t figure out in the early days was payments. Yes. Was that you?Marc: Yes. Itswyx: was a 4Marc: 0 2swyx: 0 2 4Marc: 0 2 payment required.swyx: We have a chance now. Nope. I don’t think we’re gonna figure it out. I don’t know. Like, what’s your take?Marc: Oh, I think, we’ll, yeah, no, now I think it’s gonna happen for sure.swyx: Yeah.Marc: Yeah. And there’s two reasons to example for sure. One is we actually have internet native money now in the form of crypto. Stable coins. Stable coins and crypto. And this is, I, I think this is the grand unification basically of ai, crypto, uh, is what’s about to happen now. Um, I think AI is the crypto killer app, I think is where, where this is really gonna come out.Um, and then the other is it’s just, it, I mean it’s just, I think it’s now obvious. It’s like obviously AI agents are gonna need money and it’s already happening, right? If you’ve got a c if you’ve got a claw and you wanted to buy things for you, you have to give it money in some form.swyx: I would say the adoption’s probably like 0.1% if, if that, but Yeah.Marc: Oh, today? Yeah. Yeah, yeah. But think, think forward, like where is it goingswyx: forward thinkingMarc: The ultimate principle of everything and, and everything that I think I, we, we do is, it’s the William Gibson quote, which is, the future is already here. It just isn’t distributed. Mm-hmm. It isn’t, isn’t distributed yet.My friends who are the most aggressive use users of, of, of, of open claw, just like have given their clause bank accounts and credit cards. Um, and, and, and, and, and not only have they done it. Obvious that they needed to do it because it’s obvious that they needed to be able to spend money on their behalf.swyx: Yeah. Yeah.Marc: It’s just completely obvious. And so, and again, like, so the number of people who have done that today to your point is like, I don’t know, probably 5,000 or something. Yeah. Butswyx: it’ll grow.Marc: That’s how these things startswyx: actually, I mean, since, uh, you keep mentioning,Marc: and by the way, open cloud, by the way, if you don’t give it a bank account, it’s just gonna break into your, your, it’s gonna break high agency, it’s gonna break into your bank account anyway, and, and take your money.So you, you might, as you might as well do it, you might as well do it,swyx: uh,Marc: by the way. I really love, I gotta tell you, I really love the phenomenon. I love the Yolo. Um, I’m not doing it myself to be clear, but, but I love the people that are just like, yeah, what, what is it? Skip, skip, vision,swyx: danger, skip.Marc: Dangerous.swyx: Which by the way, is a Facebook thing.Marc: Okay?swyx: Right. Because, uh, because we, uh, in Facebook, they, they have this culture to name the thing dangerous, so that you are aware when you enable the flag that you are opting into a dangerous thing.Marc: Okay, good.swyx: And they brought it into open ai and of course thatMarc: makes it enticing.swyx: Sam runs Codex, uh, with skip permissions on, on his laptop.Marc: Yes, a hundred percent. And so I, I th I think the way to actually see the future is to find the people who are doing that. There’s a man, you know, and they, you knows,swyx: log everything, you know, just watch it, watch the logs,Marc: but. Let’s actually find out what the thing can do.Yeah. And the way to find out what the thing can do is just like, try everything. Yeah. Let it try everything. Let it unlock everything. By the way, that’s how you’re gonna find all the good stuff it can do. By the way. That’s also how you’re gonna find all the flaws. Yeah. I think the people who turn that on for bots are like, they’re, they’re like martyrs to the progress of human civilization.Like, I feel very bad for their descendants that their bank accounts are gonna get looted by their bots in the first like 20 minutes. But I think the contribution that they’re making to the future of our species is amazing.swyx: It’s like gentleman science, you know?Marc: Yes. It’s, yes, yes. Experi yourself. It’s, uh, Ben Franklin out with the, trying to try, trying to get lightning to strike his, his, uh, his balloon and see, seeing if he gets electrocuted.swyx: Yeah.Marc: It’s, uh, Jonas sk with the polio vaccine, right. Injecting it. Yes. So, yes. I, I, I, I think we should have, like agl, we should have like flags and like we should have like monuments to the people that just let open club run their lives.swyx: More anecdotes of like, what, what are the craziest or interesting things that people listening to this should go, go home and do.Marc: I mean, this is, this is the, this is the, the extreme thing is just like the straight Yolo, like just Yeah. Turn, turn your lifeswyx: on. I mean, that’s a general capability. Yeah. Yeah. Is there like a specific story that was like, wow. And, and everyone in a group chat just lit up.Marc: I mean, like, you know, so there’s tons of, there’s already tons of health, you know, there’s the health dashboard stuff is just, is just absolute personal health.Absolutely amazing. Yeah. The number of stories on, um, I just don’t wanna violate people’s, you know, obviously personal. Yeah. Anonymized. But, um, you know, one of the things open clouds are really good at is hacking into all this stuff in your land. Uh, it’s really good. So, you know, internet of things. AKA internet of s**t.swyx: Yeah.Marc: Likeswyx: super insecure, but great. It’s discoverable.Marc: Yeah, it’s discoverable. O open claw is happy to scan your network, identify all the things. And then my, my, my friends who are most aggressive at this are having open claw take over everything in their house.swyx: Yeah.Marc: Take it takes over their security cameras.It takes over their, their, you know, their whatever their, their access control systems. It takes over their webcams. I have a friend whose claw watches him sleep. Put a webcam in your bedroom. Put the, put the claw, put the claw on a loop. Uh, I have it. Wake up frequently and have it watch, just tell it, watch me sleep.And, and I’ve, I’ve seen the transcripts and it’s literally like Joseph asleep. This is good. This is good that Joe’s asleep. ‘cause you know, I have, I have his health day and I know that he hasn’t been getting enough sleep and so it’s really good that he’s getting sleep. I really hope he gets his full, whatever, you know, five hours of REM sleep.Uh, Joe’s moving. Joe’s moving. Um, uh, Joe might be wake waking up. This is a real pro. If Joe wakes up now, he is gonna ruin his sleep cycle. Oh, okay. It’s okay. Joe just rolled over. Okay. He’s gone back to bed. Okay, good. Alright. Okay. I can relax. This is fine. He’sswyx: monitoring the situationMarc: monitoring, monitoring the situation, and, and being a bot, like, you know, is just like very focused, right?It’s just like, uh, this is like, its reason for existence is to watch Joe sleep. And then, and then I was talking to my friend who did this is like, you know, on the one hand it’s like, all right, this is weird and creepy. Um, and I need to, I need to, maybe this has taken over my life. And then the other thing is like, you know what if I had a heart attack in the middle of the night, this thing literally would like freak out and call 9 1 1.Like, there’s no question. This thing would figure out how to like, alert medical authorities and like, prob probably some in SWAT teams and like, do whatever would be required to save my life. Right? And so it’s like, you know, like, yeah. Like that’s happening. What else? Um, I’ll give, I, um, uh, it’s a company unitary, uh mm-hmm.That makes the robot dogs. Um, and I, I actually have one at home, which is, it’s actually really fun. The Chinese companies, the Chinese companies are so aggressive at adopting, uh, new technology, but they don’t always like, listen, take the time to really.swyx: Package it,Marc: package it, and maybe think it all the way through.And so, so the, at least the industry dog I have, so it, it has a old non LLM just control system, which by the way is not very good in, in markets. Well, but it, in practices, it’s not that good. It has trouble with stairs and so forth. And so it’s not quite what it should be. But then the language model thing comes out in the voice.So they, they add, so they add LLM capability and then they, they add a voice mode to it. Um, but, but that LLM capability is not at all connected to the control system. So, so you’ve got this schizophrenic dog that like, is a complete idiot when it comes to climbing the stairs, but it will happily teach you quantum mechanics.Right. In like a lum English accent. Right. Like, it, it, it is just like absolutely amazing. Jagged intelligence. Yeah. Yeah. Talk about jagged and then, now obviously what’s gonna happen in the future is, is they’re gonna connect together, but they’ll do it. But right now it’s, it’s, and so right now it’s not that useful.And so I, I have a friend who has one of these who had his claw basically hack in and rewrite the code Rew write new firmware. Yeah. Write new firmware for the, for the unit robot. Ooh. And now it’s, now it’s an actual pet dog for his kids.swyx: You could do that before or after like. The motion.Marc: Yeah. It’s, he said it’s completely different.He said it’s a complete transformation. Yeah. And whenever there’s an issue in the thing, now the claw just like reiterates the code. You know, you know, you goes in, it does, does the code and so is it kind of goes to your thing here. So, so like all of a sudden, uh, this is why the way we wanna think about AI code AI coding is not just like writing new apps.It’s also going in and rewriting all the old stuff that should have worked that never worked. And so, like, I, I think, I think basically, I think the internet, the internet of s**t is basically over. Like, I, I think everything, there’s a potential here where like all these devices in your house that have been like basically marginal or you know, basically dumb, you know, like all of a sudden they might all get really smart.Now you have smartswyx: home.Marc: You have to decide if, yes, there are horror movies in which this is just, of which this is the premise. And so you have to decide if you want this. Yeah. But, but, but this is the first time I can say with confidence, I now know how you could actually have a smart home. Yeah. Yeah.With 30 different kinds of things with chips and internet access, where it actually all makes sense and all works together and it’s all coherent in the, in the whole thing. And to have that unlock without a human being having to go do any of that work, like, you know.swyx: You know, I, I’m, I’m waiting for a, sorry, mark.Uh, I can’t let you open that fridge door, you know, likeMarc: Exactly, exactly. Yes, yes.swyx: Because Oh, yeah, yeah. You’re not supposed to eat rightMarc: now. I have all of, yes, I have every shred of health information, you know, and I know you think you’re doing, you know, da da da. I didn’t think you do this, but you know, this is a real, are you really, you know, are you really sure?And you know, you told, you know, you told me last night, you really don’t want me to let you do this, so, you know, I’m sorry, but the fridge door is locked. Um, yes. Openswyx: the fridge doors.Marc: Exactly. And by the way, I know you’re supposed to be studying for a test, so why don’t we, why don’t you go when you can pass the test, um, I will open the fridge door for you.Yeah.swyx: Final protocol and then, and then we can wrap up, uh, proof of humanMarc: Yes.swyx: Uh, right.Marc: Yeah.swyx: That’s the last piece that we gotta figure out.Marc: Yeah. So I would say there’s, there’s two massive, I would say, um, uh, sort of asymmetries in the world right now where we’ve known these asymmetries exist and we, we societally have an unwilling to grapple with them.And I think they’re both tipping right now. And, and they’re, they’re, they’re, they’re the same thing. It’s virtual world version. It’s a physical world version. So the virtual world version is, is the bot problem. We’re just like, you know, the internet, internet is just like a wash and bots, internet’s a wash and fake people.It has been forever. Um, by the way, a lot of that has to do with lack of money, you know? And so this, you know, this is the Yeah, this is this.swyx: My spicy take was these two are the same thing. And corporations of people too, you know? So interesting.Marc: Yeah, yeah, yeah.swyx: Okay. So a bank account is proof of human.Marc: Yeah.Okay. Yeah. Until you, until you give the bots bank accounts. Yeah, exactly. So, okay. Yeah. So there’s that. But yeah, look, look, the bot, I mean, every social media user knows this. The bot, the bot problem is a big problem. You know, the bot, the bot problem has been a big problem forever. It’s, it’s a huge problem.And it’s never really been confronted directly, like at any point, by the way. The physical world version of this is the drone, the drone problem. Um, right. And so we, we’ve known for, you know, we’ve known for 20 years now that the asymmetric threat both in Milit military and actual military conflict, but also in just like security, like, like, you know, security on the home front.The big threat is, is the cheap attack drone. Right? The, the, the cheap, the cheap suicide, you know, drone with the bomb. And we’ve known that forever. And by the way, like, you know, it’s very disconcerting how like every, you know, every office complex in the, in the co you know, in the world is like unprotected from drone attacks.Um, every, every stadium, every school, every prison. Like, like, sure e okay, we’ve known that, we’ve never done anything about what you gonna doswyx: about it. Yeah.Marc: One possibility is just leave, leave them unprotected forever and live in a world of like, asymmetric terrorism forever. Or the other is take the problem seriously and figure out the set of techniques and technologies required to, to be able to deal with that.Whether those are lasers or jammers or early warning systems, or, you know, allswyx: personal force fields,Marc: kinetic, personal for dune, uh, personal, personal force fields. Exactly. And in both cases, the, these are, these are economic asymmetries. These are economic asymmetries, right? ‘cause it’s really cheap to field a bot, but it’s very hard to tell something, a bot.It’s very cheap to field a drone. It’s very hard. It’s very expensive to defend against a drone. But you see what I’m saying is it’s, it’s, it’s the, it’s the virtual version of the problem, and it’s the physical version of the problem. Uh, the virtual version of the problem. What we, what we need quite literally is proof of human.The reason is because you’re, you’re, you’re not gonna have proof of bot. The, the, the, especially now the, the bots are too good. The, the, the bots can pass the Turing test. And if the bots can pass the Turing test, then you can’t, you can’t screen for bot. You can’t have proof of not a bot. But what you can have is you can have proof of human, you can have, you know, cryptographically validated, this is definitely a person, and this is, and then you can have cryptographically validated.This is definitely like something that a person said, yeah, this video is real. Right. Um,swyx: just to double click on, on, uh, do you think Alex Lanya with world? Yeah. Do you think he’s got it or is there an alternative?Marc: Oh, so I mean, there’s gonna be, I think there’ll be, I think many people will try, we’re one of the key, you know, participants in, in, in the World, in the World Project.I dunno that, yeah. So we’re, we’re partisans, but yeah, I, I think so we think world is exactly correct. Okay. And, and the reason is it, it has, it has to be, it, it has to be proof of human. It it has, because you can’t do proof of not bought. You have to do proof of human to do proof of human. You, you need, you need biological validation.You, you needed to start with this was actually a person, right? Because otherwise your bot signing up as fake people. Right? So you, you have to have like something, you have to have a bi. Biometric. And then you have to have cryptographic validation. And then the ability to do, to do, to do the lookup. And then, by the way, the other thing you need, which that you, you also need selective disclosure.Um, so you need to be able to do proof of human without reviewing privacy, all the underlying information. Privacy. Yeah. By the way, another thing you’re need, you’re gonna need proof of age, right? ‘cause there’s all these laws in all these different countries now around you need to be 13 or 16 or 18 or whatever to do different things.And so you’re gonna, you’re gonna need a, you know, sort of validated proof of age, um, you know, to be able to legally operate, right? And so that, that’s coming. And then you’re gonna want like, proof of credit score and, you know, proof of like, you know, a hundred other things.swyx: That’s a tricky one.Marc: It is a tricky one, but you’re gonna, you’re gonna, there, there’s no reason, like if somebody’s checking on your credit, somebody shouldn’t, I’ll give you an example.Somebody shouldn’t need to know your name in order to be able to find out whether you’re credit worthy.swyx: Right? I see. Independently verifiable pieces of information.Marc: Pieces of information, yeah. It’s like selectively disclosed. And this is the answer to the privacy problem wr large, which is, I, I only need to prove, I need to prove at that moment.So like, you’re gonna need that. And I, I think their, their, their architecture makes sense. So that needs to get solved. I think language models have tipped, the bots are now too good. Uh, and, and, and so they’re undetectable. And so as a consequence, you, we now need to go confront that problem directly. And then, and like I said, and then the other problem is we, we need to go actually confront the drone problems.The Ukraine conflict has really unlocked a lot of thinking on that. And now the, um, and now the, the, the, the, the Iran situation is also unlocking that. And so I think there’s gonna be just like this incredible explosion of, of both drone and counter drones.swyx: Our drones are better than their drones to keep it that way.Marc: Yeah. Yeah. And counter drones,Alessio: I think we can sneak in one more question. Go for it. Um, I’m trying to tie together a lot of things that you said over the years. So at the Milken Institute debate with Teal, which is amazing. Um, you talked about the lag between a new technology and kinda like the GDP, um, impact of it.Marc: Yep.Alessio: The other idea you talked about is bourgeois capitalism and how, you know, this kind of managerial class was needed because of this complexity. And I think if you bring AI into the fold, you have like much higher leverage of people. So like if you have, you know, the Musk industries, um, and you give Elon a gi, you can run a lot more things That’s right.At once.Marc: That’s right.Alessio: And then you have the social contract. And I know you reviewed a clip of Sam ing, um, we’re rethinking the whole thing, and you’re like, absolutely not. Yes.Marc: Under,Alessio: and I wa I was in an event with Sam last night, uh, and he actually said in the last couple weeks it felt like now people are taking that seriously.Yeah. So I’m just curious like how you’re seeing the structure of organization changing, especially when you invest in early stage companies and, um, yeah, just like how the impact of. Work structure and, uh, all of that is playing out. Yeah.Marc: So there’s a whole bunch of, there’s a whole bunch of topics. I know, yeah.We, we could spend, and by the way, we’d be happy to spend more time, but we could, we could spend more time on all that. So just for people who haven’t followed this, so the, this, this, this term managerial comes from this thinker in the 20th century, James Burnham, who, um, just one of the great kind of 20th century political thinkers, um, societal thinkers.And he sort of said a as, and he was writing in like the 1940s, 1950s. Um, and he said kind of the, the whole history, capitalism until that point had been in two phases. Number one had been what he called bourgeois capitalism, which was think about as like name on the door, like Ford Motor Company. ‘cause Henry Ford runs the company.Um, and Henry, it’s like a DIC dictatorial model. And Henry Ford just like tells everybody what to do. And he said the problem with bourgeois capitalism is it doesn’t scale. ‘cause Henry Ford can only tell so many people to do so many things. And then he runs at a time in the day. And so, um, he said the second phase of capitalism was what he called managerial capitalism, which was the creation of a professional class of managers, um, that are trained not to be like.Car experts or to be whatever experts in any particular field, but are trained to be experts in management. And then that led to, you know, the importance of like Harvard business, you know, business schools and management consulting firms and all these things. And then you look at every big company today, and like most of the executives at most of the Fortune 500 companies are not domain experts in whatever the company does.And they’re certainly not the founders of those, but they’re professional managers. And in fact, in the course of their careers, they’ll probably manage many different kinds of businesses. They’ll rotate around and they might work in healthcare for a while and then work in financial services and then go work in something else, you know, come work in tech.And what Burnham said is he said that transition is absolutely required because the, the, the, the problem with bourgeois capitalism is, is it doesn’t scale. Henry Ford doesn’t scale. And so if you’re gonna run capitalist enterprises that are gonna have millions to billions of customers, um, you’re gonna need to, you’re, they’re gonna be operating a level of scale and complexity that’s gonna require this professional management class.And he said, look, the, the professional management class has its downsides. Like they’re not necessarily experts at doing the thing. They’re not as inventive, you know, they’re not gonna create the next breakthrough thing. But he is like, whether you think that’s good or bad or whatever is what’s gonna be required.And basically that’s what happened. Right. And so he wrote that book originally in like 1940, you know, over the course of the next 50 years, basically. Managerialism. Well, I mean, today, up till today, managerial managerialism basically took over everything. Mm-hmm. And you know, what I’m describing is basically how all big companies run and how all governments run and how are large scale nonprofits run and kind of everything, you know, everything runs basically what, what, what Venture Capital does is we basically are a rump, uh, sort of protest movement to that.To try to find the next Henry Ford or, or just to say El Elon Musk or, or the next, or the next Elon Musk or the next Steve Jobs, or the next Bill Gays. The next Mark Zuckerberg. And so we, we, we, we start these companies in, in the old model, right? We, we, we start them out as, as, as, as in the Henry Ford model.And so we start them out with a founder or a, or a, or a founder with, with colleagues. But you know, there’s the a founder, CEO, um, and then we basically bet that we basically bet that the startup is going to be able to do things, specifically innovate in ways that the big incumbents in that industry are not gonna be able to do.And so it’s a bet that by, basically by relighting this sort of name on the door, you know, kind of thing. Mm-hmm. This new innovative thing with like a king monarchical, uh, uh, political structure, um, that they’re gonna be able to innovate in a way that the incumbent is not going to be able to because the incumbent is, is being run by managers.Right. And, and, and, and by the way, and of course venture being what it is, sometimes that works, sometimes it doesn’t. But we’re, we’re constantly doing that, but I’ve always viewed it my entire life as like, we’re like raging against the dying of the light. Mm-hmm. Like we’re, we’re, we’re, we’re sort of constantly trying to fight off managerialism, just basically swamping everything and everything.Getting basically boring and gray and dumb and old. Right. And we’re trying to keep some level of energy vitality in the system. AI is the thing that would lead you to think, wow, maybe there’s a third model.Alessio: Mm-hmm.Marc: Right? And, and maybe may and way to think about it would be, maybe it’s a combination of the two, maybe the new Henry Ford or the new Elon or the new Steve Jobs plus ai, right.Is the best of both. Right. Because it’s, it’s, it’s sort of the spark of genius of the name on the door model, the Henry Ford model. But then it’s give that person AI superpowers to do all the managerial stuff and let the boss draw the managerial stuff. That may be the actual secret formula. And we’ve never even known that we wanted this because we never even thought it was a possibility.But I mean, you know, this, what is the thing that these bots are really good, they’re really good at doing paperwork. Like they’re really good at filling out forms, right? Like they’re really good at writing reports, they’re really good at reading, they’re really good at doing all the managerial work. Like they’re amazing at it.And so, yeah, so I, I think, I think the, I a hundred percent, I think the answer, the answer very well might be to get the best, best of both worlds by doing this. And then the challenge is gonna be twofold. The challenge is gonna be for the innovators to really figure out how to leverage AI actually do this.Right? Um, and, and then, and then the, the other challenge is gonna be for the, for the incumbents that are managerial, to figure out like, okay, what does that mean? ‘cause now they’re gonna, they’re, they’re gonna be facing a different kind of insurgent competitor that has a different set of capabilities than they’re used to.And so th the, this really I think is gonna force a lot of big companies to kind of figure out innovation. EE either I say figure out innovation or die trying.Alessio: Do you feel like that structure accelerates the impact on the actual GDPN economy? If you look at Space Act? Yes. The growth is like so fast. Yeah.And like, instead of having these companies kind of like Peter out in growth and impact, they can kind of like keep going if not accelerating.Marc: Yeah, that’s for sure. The hope, um, the, the, the challenge and, and you know, and, and look, the AI utopian view is of course, of course. And, and, and that’s gonna be the future of the economy.And it’s gonna grow 10 x and a hundred x and a thousand x. And we’re entering this regime of like much higher economic growth forever and consumer cornucopia of everything. And it’s, it’s gonna be great. And I, and, and I hope that’s true. I hope that’s, that’s like the u you know, that’s the current kind of utopian vision.I hope that’s true. The problem is, it goes back again. The real world is really messy. Um, and I’ll give you an example of how the real world is really messy. It requires 900 hours of professional certification training to become a hairdresser in the state of California. Um, so it’s like 35% of the economy, something like that.You have to get some sort of professional certification to do the job, which is to say that the, the professions are all cartels, right? Yeah. And so you have to get licensed as a doctor. You have to get licensed as a lawyer, you have to get licensed as a. You have to get into a union. Mm-hmm. Um, by the way, to, to work for the government, you need to be, you, you have both civil service protections and you have public sector unions.You have two layers of insulation, uh, against ever getting fired for anything or anything. Anything ever changing. I’ll give you another example. The the dock work. The dock workers one on strike a couple years ago. Mm-hmm. ‘cause they, you know, robotics, you know, if, if you go look at a modern dock, like in Asia, it’s all robots.If you go to American dock, it’s like all still guys, dragon, dragon stuff, by by hand, the dock works. Goes on a strike. It turns out there are 25,000 dock workers working on, on, on, on Docs in America. It turns out they have incredible political power. Mm-hmm. Because it’s a, it’s, it’s one of these un unified blocks of things.They won their strike and so they got commitments from the dock owners to not implement more automation. We learned a couple things in that. So number one, we learned that even a union as small as 25,000 people still has like tremendous political stroke. We also learned that they, it actually turns out the Dock Workers Union has 50,000 people in it.‘cause there’s 20, they have 25,000 people working in the docks. They have 25,000 people during full paycheck sitting at home from prior union agreements. Oh myswyx: God.Marc: From prior union agreements. I’ll give you another great example. There are government agencies, there are federal government agencies where the employees right of have civil service protections and there are in public sector unions.There are entire federal government agencies that struck new collective bargaining agreements during COVID, where not only are they have their jobs guaranteed in perpetuity, but they only have to report to work in an office one day per month. And so there are entire office buildings in Washington DC that are empty 29 outta 30 days of the year that are still operating and are still, we’re all still paying for it.20 and say, and then what they do, it turns out what the employees do is they’re very, they’re very smart in, in, in this way. And so they figure out, they come in on the last day of a month and the first day of the next month. And so and so, they’re, so, they’re in there, they’re in the office two days per 60 days, which means these buildings are empty for 58 days at a time.And you see what I’m, you see where I’m heading with this? Like this is like locked in, right? This is like locked in in a way that has nothing to do with like, and people say capitalist, it’s like anticapitalistic. It’s like, it’s, it’s basically it’s restrictions on trade, it’s restrictions on the ability to like change the workforce.And so, so much of our economy is, is, you know, the, the, I I’m, I’m describing the entire healthcare system. I’m describing the entire legal profession. I’m describing the entire housing industry. I’m describing the entire education system, right? K through 12 schools in the United States. They’re a literal government monopoly.How are we gonna apply AI and education? The answer is we’re not, because it’s a literal government monopoly, it is never going to change the end. And there is nothing to do, by the way, you can create an entirely new school system. Like that’s the one thing you can do, is you can do what Alpha School’s doing.You can create an entirely new school system. Other than that, you’re not gonna go in and change what’s happening in the American classroom, like K through 12. There’s no chance the teachers are 100% opposed to it. It’s a hundred percent not gonna happen. So, so you see what I’m saying is like there’s this like massive slippage that’s gonna take place.Both the AI utopians and the AI dors are far too optimistic.swyx: Right.Marc: You see what I’m saying? Be because they believe that because the technology makes something possible that 8 billion people all of a sudden are gonna change how they behave. And it’s just like, nope. So much of how the existing economy works.Mm-hmm. It’s just, it. It’s just like wired in. And so we’re gonna be lucky as a society, we’re gonna be lucky if AI adoption happens quickly. Right. Because if it doesn’t, what we’re just gonna have is stagnation.Alessio: Awesome. Mark. I know you gotta run.swyx: Yeah. We all know or still welcome. But, uh, it was such a pleasure talking to you.Uh, we’re truly living in the age of science fiction coming to real life.Marc: Yes. Yes. Could not be more exciting. Yeah. Really. Thank you, mark. You guys awesome.swyx: Thank That’s it.Marc: Good. Thank you. That’s it.

We’ve been on a bit of a mini World Models series over the last quarter: from introducing the topic with Yi Tay, to exploring Marble with World Labs’ Fei-Fei Li and Justin Johnson, to previewing World Models learned from massive gaming datasets with General Intuition’s Pim de Witte (who has now written down their approach to World Models with Not Boring), to discussing the Cosmos World Model with with Andrew White of Edison Scientific on our new Science pod, to writing up our own theses on Adversarial World Models. Meanwhile Nvidia, Waymo and Tesla have published their own approaches, Google has released Genie 3, and Yann LeCun has raised $1B for AMI and published LeWorldModel.

Today’s guests have a radically different approach to World Modeling to every player we just mentioned — while Genie 3 is impressive, its many flaws demonstrate the issues with their approach - terrain clipping, noninteractivity (single player, no physics/no objects other than the player move), and maximum of 60 second immersion.

Moonlake AI (inspired by the Dreamworks logo) is the diametric opposite - immediately multiplayer, incredibly interactive, indefinite lifetime, capable of MANY different kinds of world models by simulating environments, predicting outcomes, and planning over long horizons. This is enabled by bootstrapping from game engines and training custom agents:

In Towards Efficient World Models, Chris Manning and Ian Goodfellow join Fan-Yun in explaining why their approach to efficiency with structure and casuality instead of just blind scaling is sorely needed:

SOTA models still show physical or spatial understanding glitches, such as solid objects floating in mid-air or moving “inside” other solid objects.

If the goal is to plan for the next action, how often is a high-resolution pixel view necessary for modeling the world? Our bet is that there is a disproportionately large share of economically valuable tasks where such detail is not required. After all, humans with a wide variety of sensory limitations have little difficulty doing almost everything in the world. Furthermore, for a large number of purposes, describing a scene or a situation in a few words of language (“the car’s tires squealed as it cornered sharply”) is sufficient for understanding and planning.

Experiments also show that humans only partially process visual input in a top-down, task-directed way, often making use of abstracted object-level modeling. In almost all cases, partial representations combined with semantic understanding are sufficient.

…If the goal is to facilitate the understanding of causality in multimodal environments, then the world model—whether it is used in the virtual world or the physical world—must prioritize properties such as spatial and physical state consistency maintained over long time periods, and an ability to evolve the world that accurately reflects the consequences of actions. That’s what Moonlake is building.

Game engines are the right starting point abstraction to efficiently extract causal relationships, and building the interfaces and community (including their new $30,000 Creator Cup) to kickstart the flywheel of actions-to-observations.

We were fortunate enough to attend their sessions at GDC 2026 (the Mecca of Game Devs), and were impressed by the huge variety and flexibility of the worlds people were building with Moonlake’s tools already! Live videos on the pod.

Full Video Pod on YouTube!

Timestamps

00:00 Benchmarking Gets Hard00:47 Meet Moonlake Founders01:26 Why Build World Models03:12 Structure Not Just Scale05:37 Defining Action Conditioned Worlds07:32 Abstraction Versus Bitter Lesson14:39 Language Versus JEPA Debate20:27 Reasoning Traces And Rendering Layer37:00 Gameplay Over Graphics38:02 Fiction Rules And World Tweaks39:15 Code Engines Beat Learned Priors41:10 Diffusion Scaling Limits43:23 Symbolic Versus Diffusion Boundary46:14 Platform Vision Beyond Games50:24 Spatial Audio And Multimodal Latents54:23 NLP Roots Hiring And Moon Lake Name

Transcript

[00:00:00] Cold Open

[00:00:00] Chris Manning: Think this whole space is extremely difficult as things are emerging now. And I mean, it’s not only for world models, I think it’s for everything including text-based models, right? ‘cause in the early days it seemed very easy to have good benchmarks ‘cause we could do things like question answering benchmarks.

[00:00:20] But these days so much of what people are wanting to do is nothing like that, right? You’re wanting to get some recommendations about which backpack would be best for you for your trip in Europe next month. It’s not so easy to come up with a benchmark, and it’s the same problem with these world models.

[00:00:41] Meet the Founders

[00:00:41] swyx: Okay. We’re back in the studio with Moon Lake’s, two leads. I, I guess there’s other founders as well, but, sun and Chris Manning. Welcome to the studio.

[00:00:54] Fan-yun Sun: Thanks. Thanks, Chris. Thanks for having us.

[00:00:56] swyx: You’ve got, you guys have, come burst onto the scene with a really refreshing [00:01:00] new take of mold models.

[00:01:01] I would just want to, I guess ask how you, the two of you came together. Chris, you’re a legend in NLP and just AI in, in, in general. You’re, you’re his grad student, I guess

[00:01:10] Fan-yun Sun: Actually my co-founder.

[00:01:11] swyx: Oh, yeah.

[00:01:12] Fan-yun Sun: I should give a lot of credit to my co-founder, Sharon. Yeah. She was, she was actually working with Professor Fe Androgyn and then she ended up working with, Ron and Chris Manning here.

[00:01:22] And then, so I got connected through to Chris initially, actually through my co-founder,

[00:01:26] What is Moon Lake?

[00:01:26] swyx: what is Moon Lake? What, what is, actually, I’m also very curious about the name, but like why going into world models?

[00:01:33] Fan-yun Sun: So I was working a lot. With actually Nvidia research during my PhD years on essentially generating interactive worlds to train reinforcement learning agents or embody EA agents.

[00:01:44] And then there’s two observations. One in academia and one in industry. An industry like folks at Nvidia are actually paying a lot of dollars to purchase these types of interactive worlds, whether it’s for the sake of evaluation or training the robots, or policies or models. And [00:02:00] then, in academia, same thing is happening.

[00:02:02] And more specifically, when I was actually working with Nvidia on the synthetic data foundation model training project, we were actually generating a lot of these synthetic data and showing that, hey, you can actually, these synthetic data are actually as useful as real world data when it comes to multimodal pre-training.

[00:02:16] But then, like I said, there’s a lot of dollars being paid out to like external vendors or, or like. Other folks to manually curate these types of data. It was very clear to us that, okay, on our way to, let’s call it embody general intelligence models need to learn the consequences behind their actions, which means that they need interactive data and the demand for those types of data are growing exponentially.

[00:02:38] But everybody’s sort of thinking about it from a pure, say, video generation perspective or something else. But we feel like the true actually opportunity is actually building reasoning models that can do these things, like how humans do these things today. So that’s a little bit on the genesis of Moon Lake, and I think the reason I got into world models was partly.

[00:02:59] A philosophical [00:03:00] take of the on the world where I like, believe the simulation theory and stuff like that. But on the other, on the other hand, it’s really just like, oh, like there’s an opportunity there that I feel like nobody’s doing it the way I think should be done.

[00:03:10] Structure, Not Scale: The Vision

[00:03:10] Chris Manning: I can say a little bit about that.

[00:03:12] Yeah. So of the overall goal is the pursuit of artificial intelligence and most of my career has been doing that in the language space and that’s been just extremely productive. As we all know, the story of the last few years, I don’t have to tell about how much we’ve achieved with large language models, but, uh.

[00:03:31] Although they have been extremely effective for ramping language and general intelligence, it’s clearly not the whole world. There’s this multimodal world of vision, sound, taste that you’d like to be dealing with more than just, language. And then the question is how to do it. And despite, a huge investment in the computer vision space, right, as the research field computer [00:04:00] vision has been for decades, far, far larger than the language space, actually.

[00:04:05] I think it’s fair. Say that, vision, understanding sort of stalled out, right? You got to object recognition and then progress just wasn’t being made right? If you look at any of these, vision language models, it’s the language that’s doing 90% of the work and the vision barely works. And so there’s really an interesting research question as to why that is and at heart, the ideas behind Moon Lake are an attempt to answer that, believing that there can be a really rich connection between a more symbolic layer of abstracted understanding of visual domains, which aren’t in the mainstream vision models, which are still trying to operate on the surface level of pixels.

[00:04:50] swyx: I think one of your blog posts, you put it as structure, not scale. Is that, a general thesis?

[00:04:57] Chris Manning: Yeah. Well, scale is good too.

[00:04:58] swyx: Yeah. Scale is good. Too

[00:04:59] lot,

[00:04:59] Chris Manning: [00:05:00] lots of data is good as well and scale, but nevertheless, you want the structure Yeah. To be able to much more efficiently learn.

[00:05:07] swyx: Yeah. The other thing I really liked also is you put out an example of what your kind of reasoning traces look like.

[00:05:12] Right. Which you would distill is the word that comes to mind. I don’t even think that’s a good, good description, but it would involve, for example, geometry, physics, affordances, symbolic logic, perceptual mappings, and what, what have you. But like that, that is the kind of example that involves, let’s call it spatial reasoning, role model reasoning as as compared to normal LM reasoning.

[00:05:35] Yeah.

[00:05:36] Defining World Models vs Video Generation

[00:05:36] Vibhu: But also like taking it a step back. So how do you guys define world models? A lot of people see okay, you can do diffusion, you can do video generation. But, you guys put out quite a few blog posts. You put out a essay recently, we can even pull it up about efficient world models. You have a pretty like structural definition here, but for the general audience that don’t super follow the space, right.

[00:05:55] What’s, what’s the difference in what we see from like a video generation model to [00:06:00] a world gen A simulator? How do you kind of paint that last

[00:06:02] Chris Manning: year? Yeah, so I think this is actually a little bit subtle because, people look at these amazing generative AI video models, SAWA VO three, one of these things, and they think Genie, they think, oh, this is amazing.

[00:06:17] This is we’ve solved understanding the world because you can produce these generative AI videos, but. The reality is that although the visuals do look fantastic, those visuals actually are accompanied by an understanding of the 3D world, understanding how objects can move, what the consequences of different actions are, and that’s what’s really needed for spatial intelligence.

[00:06:49] So I mean, a term we sometimes use is that you need action condition, world models. That you only actually have a world model if you can predict, [00:07:00] given some action is taken, what is going to change in the world because of it. And in particular, that becomes hard over longer time scales. So if you’re simply, trying to.

[00:07:12] Predict the next video frame. That’s not so difficult. But what you actually want to do is understand the consequences, likely consequences of actions minutes into the future. And to do that, you actually much more of an abstracted semantic model of the world.

[00:07:32] The Bitter Lesson & Data Abstraction

[00:07:32] swyx: Yeah, the question comes where you want to have more structure than is available in just predicting the next token.

[00:07:41] And typically, well, let’s, let’s call it the experience of the last five years has been that is just washed away by scale, right? So what is the right middle ground here that, you don’t ignore the bitter lesson, but also you. Can be more efficient than what we’re doing today.

[00:07:57] Chris Manning: One possibility [00:08:00] is, look, if we just collect masses and masses and masses and masses of video data, this problem will be solved.

[00:08:11] Under certain assumptions that could be true, but there are sort of multiple avenues in which it could not be true. The first is what’s really essential is understanding the, the consequences of actions producing an action conditioned world model. And if you are simply, collecting observational video data, which is the easy stuff to collect, when you’re sort of mining online videos, you don’t actually.

[00:08:41] Know the actions that are being taken to see how the video is changing. And so if you are never collecting directly actions and you are having to try and infer them from what happened in the observed video, that’s not impossible. But it’s very [00:09:00] hard and it’s not really established that you can get that to work at any scale yet.

[00:09:05] And so there’s a lot of premium on collecting action condition video data, which is part of why there’s been a lot of interest in using simulation so that you can be collecting data where you do know the actions, which isn’t quite limited supply, but there’s also in the limit of as much data as you could possibly have.

[00:09:28] Maybe the problem is eventually solvable, but. Even though we collect huge amounts of text data is always at a great level of abstraction, right? Language is a human designed, abstracted representation where there’s meaning in each token and it’s representing and abstraction of the world, right?

[00:09:51] As soon as you are describing someone as a professor, and as soon as you are saying that they’re condescending, right? These are very [00:10:00] abstracted descriptions of the world. It’s not at what you’re observing as pixel level, and to get to that kind of degree of abstraction, starting from pixels is orders and magnitude of extra data and processing.

[00:10:14] And so, although, we absolutely want to exploit, get as much data as possible, use the bitter lesson. Nevertheless, if there are ways in which you can work with five orders of magnitude less data than people working purely from pixels, you’re gonna be able to make a lot more progress, a lot more quickly.

[00:10:34] And that’s the bet here. And so you could just say that’s only wanting to be able to, do it more efficiently, do it more quickly, do it more cheaply. But I think it’s actually more than that, I think. One should be making the analogy to how human beings work at one level. You know? Yes, we have these high [00:11:00] resolution eyes and we can look and see a scene like a video, but all of the evidence from neuroscience and psychology is that most of what comes into people’s eyes is never processed.

[00:11:13] Right. That you are doing fairly fine ated processing of exactly what you’re focusing on. But as soon as it’s away from that of yeah, there’s another guy over there that you’ve sort of only processing top down this very abstracted semantic description of the world around you. And so, that’s what human beings are doing.

[00:11:33] They’re working with semantic abstractions and so. I think it is just the right representation. ‘cause we also have other goals we want to be able to do, real time worlds. So that means there’s a limit to how much processing you can do and we want to do long-term planning and consistency. And again, that favors abstraction.

[00:11:55] I mean, I guess there was actually a recent. Blog posts that [00:12:00] came out from our Friends of physical intelligence and, they were sort of heading in the same direction they were saying Oh, to the pay

[00:12:06] swyx: pay model.

[00:12:07] Chris Manning: Yeah. Yeah. To maintain a long term memory of what’s happening in the world. So we can, do longer term we actually storing text of what is, been happening in the world.

[00:12:19] Right. It is not such a successful strategy of trying to keep it all at a pixel level.

[00:12:24] Vibhu: And yeah, I mean, you can see it in video models like that Temporal consistency. We’re at a scale of train on, all the video data we have. We have it for maybe 30 seconds, a few minutes. That’s not the same as a game state played for half an hour.

[00:12:37] Right. I thought you guys break it down pretty well. You have a, you have a blog post about. Building multimodal worlds with an agent. I dunno if you guys wanna talk about this. This is one of the things I read, I

[00:12:48] swyx: thought, yeah, it’s the thing I talked about with the reasoning chain. Yeah.

[00:12:51] Vibhu: So there’s like different phases to this.

[00:12:53] It seems like it’s more of an agent, a scaffold, very different approach than just, type in a prompt and you, you don’t have the same consistency. [00:13:00] It also, like, for people that are listening, I, I would highly recommend reading it. It breaks down the problem in a different light, right?

[00:13:06] So like, what do you need to consider when you’re talking about video, like world game models, right? How would, what do you need to consider? What are the factors? What are the elements? What’s the state? So I don’t know if you guys have stuff to talk about for this one.

[00:13:19] Fan-yun Sun: Yeah. Actually, I wanted to add on a little bit Yeah.

[00:13:22] On our previous point, which is just like, change topics so quickly. I, I do feel like sometimes people confuse like, oh, like we’re taking an an, an method with abstraction. That means they don’t believe in bitter lesson. Like that’s just false, right? Like we are believed is a bitter lesson. But then I feel like the question that we always discuss is like, what is the right abstraction level today?

[00:13:42] The analogy I like to make is like, let’s just say we can encode and decode. Represent all of images, videos, audio and bytes. Then the most bitter lesson approached is to train a next byte prediction model as opposed to the next token prediction model where it’s just like, okay, it’s natively multimodal, can just, but it’s like, yeah, like [00:14:00] to, to Chris’s point, it’s like the scale and computing you need to achieve that.

[00:14:03] So that’s why we always come back to like, okay, what is the most efficient way to do it? And reasoning models to the point of this blog post is a showcase of like, Hey, we’re actually just like reasoning about the world and reasoning about. The aspects of the world that CAGR that matter for me to learn what I want to learn from this role model.

[00:14:21] swyx: Yeah, it’s like you’re improving the en encoder of whatever you’re, trying to model. And like a better representation would just represent the important things in less space. Yeah. Which would just be more efficient.

[00:14:33] Fan-yun Sun: Yeah.

[00:14:34] swyx: So yeah, I, I, I fully agree that it is not, antagonistic to, bitter lesson.

[00:14:38] I do wanna wanna mention one more thing. Is there any philosophical differences with the JPA stuff that, Yun is working on? I gotta go there. You, you, you, you’re, you’re imagining like some latent abstraction. I’m like, okay, fine. Let’s, let’s talk about it, right? Like it’s an elephant in the room.

[00:14:52] Chris Manning: Yeah.

[00:14:53] JEPA & Philosophical Differences with LeCun

[00:14:53] Chris Manning: There are philosophical differences. Jan Lacoon is a dear friend of mine, but. [00:15:00] He has never appreciated the power of language in particular, or symbolic representations in general. Yarn is a very visual thinker. He always wants to claim that he thinks visually and there are no words, symbols, or math in his head.

[00:15:21] Maybe that’s true of yarn. It’s certainly not the way I think. Um. But at any rate, the world according to yarn is the basic stuff of the, the world and of intelligence is visual and language is just. This low bit rate communication mechanism between humans and it doesn’t have much other utility and it’s far inferior to the high bit rate video, that comes into your eyes.

[00:15:53] And I think he’s fundamentally missing a number of important things [00:16:00] there. Think of this evolutionary argument looking at animals, right? That the closest analogies, the things with chimps, right? So chimpanzees, have fairly similar brains to human beings. They have great vision systems, they have great memory systems.

[00:16:18] They’ve got, better memory than we do of short term memories. They can plan, they can build primitive tools that, humans. Massively ahead in what we understand about the world, what we can plan, what we can build. And essentially what took off for us was that humans managed to develop language and that gave a symbolic knowledge, representation, and reasoning level, which just, okay if this sort of vaulting of what could be done with the intelligence in brains.

[00:16:59] So the [00:17:00] philosopher Dan de refers to language as a cognitive tool and argues that, humans unique among the creatures in the world have managed to build their own cognitive tools and language is the famous first example. But other things like, mathematics and programming languages are also cognitive tools.

[00:17:21] They give you an ability to. Think in abstractions, in extended causal reasoning chains. And that allows you to do much more. And we use that for spatial representation and intelligence and planning and gameplay as well. So we believe, and this is, underlying the specific technologies that Moon Lake is making, that symbolic representations are powerful.

[00:17:50] And you want to use that in your understanding of the visual world when you want a causal understanding, when you want to maintain long-term [00:18:00] consistency and prediction. And as I understand it, that’s just not in ya Koon’s worldview. So I think that’s the fundamental philosophical difference. Then there’s the specific model.

[00:18:11] He’s been advancing jpa, that’s a reasonable. Research bed is a direction as to, to head for building out a model of the visual world. To my mind, it’s sort of one reasonable research bed. It’s not really established. It’s the best one that everyone should be following,

[00:18:32] swyx: at least developed at scale, at Meta.

[00:18:34] But it’s not just vision, right? Like, I mean, JPA is a, just joint admitting prediction can be applied to anything really. And people have done it. The argument is that there is a latent representation or that is probably more. Suited to the task, then why not let machines do it for us instead of predefining it at all?

[00:18:50] And isn’t something like a JPA shaped thing the right answer? And if not, why not?

[00:18:55] Chris Manning: So I think there’s a part of jpa that’s right, which is [00:19:00] you do want to have a joint. Embedding that gives you a consistent model of the world. And Jan’s argument is you can never get that from auto aggressive language models ‘cause they’re sort of left to right churning out one token at a time.

[00:19:22] I guess this is where we’re the research arguments of the field, I’m not actually convinced that’s right. ‘cause although the token production is this auto aggressive, process that’s heading, left to right, I guess don’t have to be left to right. But anyway, in sequence of tokens we could have right to left Arabic.

[00:19:40] But although that’s true, all of the weights of the model that are internal to the transformer, they are a joint model of the model’s understanding of the world. And so I think you can think of the weights of the model as a form of. Joint representation, [00:20:00] and therefore it is plausible to think that could be the basis of a world model, which avoids, ya’s objections.

[00:20:10] swyx: I think I follow, and obviously that would touch on what Moon Lake eventually ends up doing as well. Right. Like, which it’s hard to tell because you put out the end results, but we don’t know the inputs that go into it. So it’s, it’s, that’s something that we have to figure out over time.

[00:20:25] Vibhu: Yeah. I mean, I guess this kind of breaks down some of the outputs. Do you wanna walk us through it?

[00:20:31] Reasoning Traces & Interactive Worlds

[00:20:31] Fan-yun Sun: Yeah. So this, this really just walks us through the reasoning traces of like, okay. So that just say, if we wanna build a world in this context, it’s really just a game demo that, that shows the, the variety of interactions that this world model can build.

[00:20:45] And yeah, it’s really just a reasoning traces of like, okay it prompted to create a bowling game. Like how did it achieve what you saw? That level of causality, interaction and consistency, right? So yeah, this is almost just like a, an example of [00:21:00] like a reasoning traces. Very

[00:21:01] swyx: detailed.

[00:21:01] Fan-yun Sun: Yeah.

[00:21:01] Vibhu: Very, very detailed.

[00:21:02] You gotta you don’t even realize it, right? Like when a video is generated, what happens when a ball strikes a pin, right? So first, like you, there’s audio in that, like audio triggers happens, score increments, the world changes. Like pins have to start dropping. There’s a timer that goes on. It’s just like very similar to how now we’re used to reasoning for language models.

[00:21:20] There’s a whole state of what happens. So geometry, physics, all this stuff. And then yeah, there’s kind of that single prompt. So asset, ation all this stuff. It’s like a, it’s a nice view to see what’s going on.

[00:21:32] swyx: I think Sun is also too polite to point out that, both like Google’s genie, demos as well as world Labs is marble, do not have interactive worlds.

[00:21:41] Fan-yun Sun: That’s the benefit of having a reasoning model, right? Like, because you can, you can say, oh, like maybe in this particular context, I want to learn how to bowl. And then you can say, okay, then what is it important when it comes to learning how to bowl? Okay, maybe it’s like I need to understand the, the basic of like, physics and I want to throw it over [00:22:00] them.

[00:22:00] I wanna know that when I, when it resets it’s a new game. So I know that yeah, basically, you know to pick up the ball, you know that ball’s gonna cause the pins to fall down. You know that what’s important to this particular bowling game is to score and you know that the score corresponds to the number of pins that fell down.

[00:22:19] So it’s just like, if it’s a model that sort of knows what it. Looks like, knows what a bowling game looks like, but doesn’t actually allows you to practice over and over again and to understand that, oh, like what it takes to actually get a high score. Then it sort of doesn’t actually allow you to learn what you set out to learn within the world model.

[00:22:38] And I think this is really just one example of showing like the advantages of the approach that we’re taking over most the, let’s call it the zeitgeist, is today, when people talk about clinical role models,

[00:22:51] Chris Manning: right? So it sort of seems like the question to ask when there’s a world model is.

[00:22:58] Can I not [00:23:00] only just wander around the world and look at the beautiful graphics, can I interact with the objects in the world and see the right consequences of actions?

[00:23:11] Vibhu: And you also understand what the consequences would be if you do something right. So it’s not just like, okay, there’s one thing if I pick it up, something will happen.

[00:23:19] But, there’s 50 options and I know I can expect, I can infer what would happen if I do any of them. Right. So very different when you can actually see it play around with it.

[00:23:28] swyx: There,

[00:23:28] Beyond Unity: Cognitive Tools for World Building

[00:23:31] swyx: there’s two cheeky elements of that. I mean, the, the, the I guess, less ambitious one is, let’s really establish for listeners, why is this fundamentally different than writing Unity code, right?

[00:23:40] Like just creating a model to translate a prompt into Unity code

[00:23:44] Fan-yun Sun: so there is an underlying physics engine. Yeah. In that sense, there’s some overlapping things to Unity, but the way we think about it is like physics engine. Tools or code are cognitive tools like borrowing Chris’s term, right? Like tools [00:24:00] that the model can employ as means to an end.

[00:24:04] So today maybe you say, okay, in this particular context we care about physics, we care about the long-term causality consequences. Then yes, we deploy it, employ physics engine, and then maybe tomorrow we say, okay, we’re we’re training that. Just say drones where we only care about really fluid dynamics and the visual aspect of the world.

[00:24:25] Then, then yeah, maybe we don’t actually, the model actually doesn’t have to use a physics engine. Or maybe it employs other types of representation or physics engine to achieve the task. So yes, writing code for Unity is sort of similar to a tool that our A model can employ, but our goal is for a model to take a representation conditioned reasoning.

[00:24:46] Approach or process.

[00:24:47] swyx: Yeah,

[00:24:47] Fan-yun Sun: internally.

[00:24:48] swyx: Yeah. Using these things as just like general two calls. Right. Which I think is very interesting. The other more ambitious one is, some kind of recursive element where it becomes multiplayer, right? Like here, there’s a single player element, you’re not [00:25:00] modeling any other people involved.

[00:25:01] And that is a whole other thing.

[00:25:04] Fan-yun Sun: But in fact, we can really do multiplayers. Oh yeah, okay. I haven’t seen any double situations. So just actually just like prompt our, our model to say, Hey, like configure to multiplayer. Then it’ll do like this. You’ll be able to configure multiplayer

[00:25:16] swyx: great

[00:25:17] Fan-yun Sun: persistency database for you.

[00:25:18] Easy. Yeah.

[00:25:19] Vibhu: So what, what are like some of the current limitations in where we’re at? So there’s one approach of like, okay, scale up video predictors. Obviously there’s data issues. With approaches like this, is it data constraints? What are like the next steps? Is it real time? Like, so there’s one side of, write an agent to write Unity code, but okay, I want to be streaming a game real time.

[00:25:38] I want to have characters being also like agent, but where, where do we kinda see this scaling up? Right?

[00:25:44] Fan-yun Sun: Yeah, there’s definitely a data constraint. Like the more data, the, the better. This reasoning model can almost basically act as humans to like operate a variety of tools and softwares to build whatever’s necessary.

[00:25:57] And then there’s a sort [00:26:00] of fidelity constraint, which we’re actually solving with another model, which we can talk about later. But it’s like, it’s not as easy to get to photorealism with the approach that we’re taking. But we think there are better solutions to that, which is we can dive into later.

[00:26:14] Later.

[00:26:15] Vibhu: The one one thing you note here is it’s a diffusion model, right? So there’s, there’s a few approaches, diffusion caution, splatting, yeah, so Ry diffusion model, you guys wanna

[00:26:25] Fan-yun Sun: Yeah.

[00:26:25] Vibhu: Introduce,

[00:26:26] Fan-yun Sun: yeah, totally.

[00:26:26] Rie: Neural Rendering & Skins for Worlds

[00:26:26] Fan-yun Sun: So within our world modeling framework, we think there are two models that we train, right?

[00:26:31] Like, there’s the multimodal reasoning model that we just talked about that essentially handles. Mainly the, the causality, the persistency and logic determinism of the world. And then RY is our bet on saying, okay, like while all those model, can take care of all these things that we just talked about, it’s limitations compared to existing, say, video models, is that it doesn’t have as high of a pixel [00:27:00] ality right off the gate, right?

[00:27:02] And EE is to say, Hey, we can actually take whatever persistent representation that we generate with our multimodal reasoning model and learn to restyle it into photo photorealistic styles or arbitrary styles you want. So this model is almost to say, Hey, I’m going to respect the persistency and interactivity of the world that you created, but my only job is to make sure that its pixel distribution is close to what we want.

[00:27:29] Vibhu: Yeah.

[00:27:30] swyx: Great example right there. You kept the KL divergence.

[00:27:33] Fan-yun Sun: Oh. Where,

[00:27:34] swyx: no, no. I mean this, this is a, a classic like, how you don’t stray too far from the source material as you, you kept the kl, which is Oh yeah. Kind of cool. Yeah.

[00:27:43] Fan-yun Sun: Yeah.

[00:27:44] swyx: I mean, and the

[00:27:44] Chris Manning: difference is, and I mean sun was pointing at this, where sort of saying it’s in one way a more difficult path, but a better path that, typically the diffusion models are producing the whole scene and it looks lovely, [00:28:00] but there isn’t spatial understanding behind it, which is allowing for the real time graphics gameplay, the spatial intelligence, understanding the consequences of worlds where this is, taking a path where it is assuming an abstracted semantic model of the world’s state.

[00:28:20] And then the diffusion model is then being used on top of that to produce the high quality graphics.

[00:28:27] swyx: Is there an intended practical, or business use for this, or is it like a, like a demonstration of capabilities?

[00:28:34] Fan-yun Sun: We actually believe that this is gonna be the next paradigm of rendering. So it’s gonna replace how ra raizer, it’s gonna replace DLSS today because it not only has these pixel prior that’s learned from the world such that you can literally play any game in photo realistic styles, which is a lot of people’s desire when they do GTA, right?

[00:28:51] Like,

[00:28:51] Vibhu: all the mods, all the people adding perfect lighting and all this.

[00:28:54] swyx: So

[00:28:54] Fan-yun Sun: skins

[00:28:55] swyx: for worlds, let’s call it

[00:28:56] Fan-yun Sun: skins, let’s call it skin for worlds. I,

[00:28:58] Vibhu: it’s also like, you can call it skin, you can call it [00:29:00] customization. You can play it how you want, right?

[00:29:01] Fan-yun Sun: Yeah, exactly. And I think another thing that we really pointed out specific specifically in this blog is the programmability of it, right?

[00:29:09] So what this means is that this render historically render is always a derivative of the game state, right? You’re saying, oh, here’s the game state, I’m rendering out a frame. But here I’m saying actually this render can be part of the gameplay loop. I can say something along the lines of, if upon getting 10.

[00:29:26] Apples, I’m gonna, my weapon of choice, my bullet’s gonna turn into apples. And that’s, that’s possible because we can say, we can basically dynamically have certain game state trigger the, the preconditions to the render such that the rendering is now part of the game loop too. One thing is to just say, okay, it’s, it’s, it’s the appearance.

[00:29:47] But the second thing is also to say there’s these novel interactions that are possible because this render now has actually priors of the world.

[00:29:57] swyx: It is up to the artist to figure out what to do with it.

[00:29:59] Fan-yun Sun: It [00:30:00] is up to the creators. Yes.

[00:30:01] swyx: Yeah.

[00:30:01] Fan-yun Sun: And I also think that’s actually another big argument that we’re making and the reason that we’re picking, taking the bet we’re baking is that a lot of the times, whether it’s for embody AI gaming, like you want a layer where human can inject their intentions.

[00:30:15] So, for example, let’s just say in the context of gaming, it’s obviously like my creative intent, but maybe in the context of embodied ai, it’s like, oh, like I take this foundational policy and I want to actually fine tune it to deploy in my house. So you want to almost say, inject, have a layer where human can say, oh, here’s the distribution of things I want to create to achieve my goal.

[00:30:35] And I think 3D graphics as it as it is today, is basic, the layer for people to say, Hey, what do I care about in this world? And it allows, basically human intent to be expressed in these worlds much more explicitly and distributionally as opposed to just saying, Hey, I’m gonna generate like, arbitrary.

[00:30:54] And it’s like just prompts,

[00:30:55] swyx: it’s one of those things where like, I think you, you’re going to build up a series of models, right? [00:31:00] This is just one of, this is probably like the highest utility or heaviest, frequency one, I don’t dunno what to call this. Where like you Yeah. You can immediately drop this in on any game and you don’t need anything else that.

[00:31:10] That you guys do. But, I, I could see, I could see that I think the, the human intent is something that people are not even used to because we’re so used to static worlds or, worlds that just don’t react, or, I don’t know. It’s, it, you’re kind of blowing my mind right now with like, I’m, I wonder if you’ve talked to people at GDC Hmm.

[00:31:27] And what are they gonna do with it?

[00:31:30] Fan-yun Sun: Yeah. Now the stance that we take on this front is like, we’re not gonna be more creative than our users to ship

[00:31:35] swyx: it out.

[00:31:35] Fan-yun Sun: Yeah. But we wanna make sure that we’re building things in a way that really allows them to express their intent.

[00:31:41] swyx: The thing that you said about, here’s the distribution that I want.

[00:31:45] I think text may be too low of a bandwidth to. To really demonstrate, because I, I, there, I’m, I’m probably just gonna want to drop in a bunch of, reference assets and then you can figure it out from

[00:31:58] Vibhu: there. But you probably wanna do a, a mixture of [00:32:00] both, right? Like you throw in a few images. I wanted this style.

[00:32:02] Yeah. I want it to look like this. So it, it’s, it’s a mixture, right?

[00:32:05] Chris Manning: I, I think it’s a mixture. I mean, yeah, I mean there’s clearly a visual component of this, and it’s not that, everything can be text. ‘cause of course you want to give a visual look, but there’s also a massive amount of giving the overall picture of the look of the world and the behavior of things that you can express in a few words of text.

[00:32:32] And it be very time consuming and difficult to do via visual means. So I think, yeah, you want a combination of both.

[00:32:40] Evaluating World Models

[00:32:40] Vibhu: So one question I kind of have is, how do we go about evaluating world models? So like, there’s many axes, right? One is like, okay. I have preferences. How well do we adhere to prompts? One is the simulation.

[00:32:50] One is like do things, is there core logic that’s broken? So coming from we know how to evaluate diffusion, there’s fidelity, there’s [00:33:00] stuff like that. But what are some of the challenges that most people probably aren’t thinking about?

[00:33:04] Fan-yun Sun: Yeah, I think this is like a great question and probably one of the hardest questions in role models because like, I think it always comes back to what are you building this role model for?

[00:33:13] And depending on your end goal and purpose, the evaluation should defer. So in the context of games, then the most direct way of measuring is how much behind are people actually spending in this world that you create? And if your goal is to say, for example, in the context that we just talked about, like, hey, deploying, deploying action in body, a agent, then your, your end.

[00:33:33] Metric is then, okay, after training in these worlds that you generate how robust it is to when you actually deploy to the target environment. But then, it’s, it’s hard to measure these end metrics. So today people have like these proxy metrics that I call that basically try to measure what we really care about, which is the end metrics, but then frankly it’s different for every use case.

[00:33:57] Yeah,

[00:33:57] Vibhu: which seems like quite a challenge, right? Like in [00:34:00] in language models or video models. Image models, your benchmarks are proxies, right? People aren’t actually asking instruction, following tool use questions. They’re proxies of how well it will do downstream. But for this, so like, should teams, should companies have their own individual benchmarks outside of games?

[00:34:16] If you think of stuff like, okay, video production, movies, stuff like that, that also want to use world models. Should, should they sort of internalize like. Their own proxy. Is this something you guys do? Where, where does that connect

[00:34:28] Chris Manning: go? Yeah, I think this whole space is extremely difficult as things are emerging now.

[00:34:35] And I mean, it’s not only for world models, I think it’s for everything including text-based models, right? ‘cause in the early days it seemed very easy to have good benchmarks ‘cause we could do things like question answering benchmarks and could you answer the question based on these documents and the various other kinds of, do pieces of logical reasoning or math.

[00:34:58] But again, these are sort of. [00:35:00] And there were sort of visual equivalents of things like object recognition, right? For these small component tasks. These days so much of what people are wanting to do also with language models is nothing like that, right? You’re wanting to, have an interaction with the language model and get some recommendations about which backpack would be best for you for your trip in Europe next month.

[00:35:25] And it’s not the same kind of thing, right? And it’s not so easy to come up with a benchmark as to does this large language model give you an effective interaction for guiding you in a good way for shopping, right? So, and it’s the same problem with these world models. So if we take the game design case, well success is that a game designer can.

[00:35:57] Produce what they are [00:36:00] imagining in a reasonable amount of time. And that’s really the kind of macro task. That’s a very hard thing to turn into a benchmark and I think a lot of this is actually going to turn into people walking, walking with their feet. Right? I mean, I guess that’s what’s happening, at the large language model level, right?

[00:36:23] When people are choosing to use, GPT five or Gemini or clawed, individuals are trying out these different models and deciding, oh, I like the kind of answers that GT five gives me, or no, I feel like I get more accurate detail from Claude, right?

[00:36:43] Vibhu: It’s a lot of

[00:36:43] Chris Manning: vitech, a lot of people just using it.

[00:36:45] It’s vibe checking. I realize that, but it’s actually whether. People feel it’s giving them utility in what they want. Right.

[00:36:52] Vibhu: And the the interesting thing there is like a lot of people prefer the visual, right? This looks pretty, which is not the objective of what this is [00:37:00] for, right? It’s if a, if a game designer is working on something, they care about the game engine, right?

[00:37:04] The state, it’s, it can look whatever. You can fix that up later. Or you can have a really good game state and you can quickly edit it to 20. 20 different versions, like Keep State,

[00:37:14] Chris Manning: right?

[00:37:14] Vibhu: So

[00:37:14] Chris Manning: that’s a really important distinction, for and for speaking to Moon Lake strength, right? So, yeah, great visuals are lovely to look at for a few seconds, but gains are really all about the concept, the game play.

[00:37:33] And a lot of the time that doesn’t actually even require great visuals. I mean, there are just lots of very successful games which have relatively primitive visuals, and there are other games where people have spent millions producing photo realistic, visuals, and the game sucks, right? So, keeping those two axes apart is really important in thinking about what’s important in a [00:38:00] world model for different uses.

[00:38:02] swyx: This conversation is reminding me of some game review and fiction discussions I’ve, had in my sort of non-AI related life. Some, for some people might know Brandon Sanderson, who’s a very famous, fiction author, had, is is a big game reviewer. And he, he’s a big fan of video games where you change one thing about a normal what you might assume about, about the world.

[00:38:22] For example, Baba is you, I don’t know if you might have come across that, where like the rules change as you play the game. And also like where, you can do things like reverse time selectively or like change gravity selectively. And I think this is also reminds, reminds me of other kinds of world models that are created by authors.

[00:38:38] Where Ted Chang is, is my typical example where he’ll take the world that, you know today, but change one thing about it and, but then create a consistent world based on that. Which is long-winded answer of me to, of. For me to say is it’s it easy to create alternative roles that don’t exist, but you change one thing and then let’s, let’s run a whole bunch of people through it to see if it works.

[00:38:58] Chris Manning: My first dance will [00:39:00] be, that seems a lot easier and more conceivable to do using Techn technology like Moon Lakes than with some of the other world models out there, where the sun can actually make it happen. I’ll let him give a second answer.

[00:39:15] swyx: If I guess for you, you’re constrained by the game engine tool, right?

[00:39:18] Like at the end of the day, that’s the, that’s the thought, partner that you have. If I ask for something where like, if it never is allowed to reverse time or if gravity only ever works one way, then well that’s it. But sometimes gravity might change,

[00:39:33] Fan-yun Sun: but it’s a lot easier to change with code as opposed to a model that is learned primarily on data of.

[00:39:42] Real world and virtual worlds that are, I guess, like for example, junior, like there’s actually trained on a lot of real world data and a lot of virtual gaming data, and it’s hard to say maybe it’s easier to say, okay, I wanna change the visuals in like the time period of, of the world. Like, you can’t change gravity, for [00:40:00] example.

[00:40:00] Vibhu: I feel like you can to light bounds, right? Everything comes down to like, code is a better way to execute it, but the models aren’t that diverse and creative, right? You can say, okay, make gravity slower. It can do that, but it’s limited to your representation of how you text it out, right? Like they’re, they’re only gonna do a few iterations, whereas programmatically, if there’s a game engine under the hood, you can kind of go wild, right?

[00:40:22] So one of the, I dunno, one of the limitations of most models is that they’re very overtrained to one style. Right. And extracting diversity is pretty difficult. At least that’s something we’ve seen.

[00:40:35] Fan-yun Sun: I mean, are there examples you have in mind where you Existing models? Yeah. Like it would be easier to do that’s not using code.

[00:40:43] Certain types of creative intent or like transition state transitions,

[00:40:47] swyx: Clipping, other models, other wo models are very good at clipping through things. Clipping my, my, my legs clipping through a rock because it’s, it’s just, it’s just bad. [00:41:00] Like, you would have to struggle very hard with your stuff to actually make that happen.

[00:41:04] Which I think is maybe a topic that you actually prepared on, Gian Splatting versus, the other stuff.

[00:41:09] Vibhu: Yeah. Yeah. It’s just for those not super familiar, right? There’s a, there’s gian splatting, there is diffusion. Like what works, what scales up. I feel like in February when Soro one came out the blog post was literally titled like,

[00:41:21] swyx: you bring it up.

[00:41:22] You never know.

[00:41:23] Vibhu: World, world, video generation models are world simulators. It’s super bitter lesson pilled. Yeah, emer, a lot of it is emergence, right? So, not to go through their blog post, basically their whole thing was as you scale up all this consistency, all this stuff just kind of solves, it’s a very simple premise, right?

[00:41:41] They just scaled up, diffusion, and from there, this is, this is Feb 2024, how much can we, it’s already been two years, which is basically five years. How much more in AI time do we need to just scale up or, or do we hit a data cap? But I think we already talked about this a lot, right? Like this is back to the beginning discussion of what’s [00:42:00] appropriate for the time.

[00:42:01] And that seems like your approach, right?

[00:42:03] Fan-yun Sun: Yeah. The point I’m trying to make is that they’re very many, many different types of world simulators and like having a world simulator that can produce pixel coherency is very, very useful for games and, marketing and all these things, but it’s not as useful as people think when it comes to causal reasoning.

[00:42:25] When it comes to embodied ai. Yeah, like it this title is true. We’re not saying that it’s, it’s like, not a great world simulator, but actually in the blog that we, we, we, we wrote, the bet is more so that there are gonna be disproportionately large share of value of real world tasks or, and virtual tasks where high resolution pixel fidelity is not needed.

[00:42:47] Yes. Video models have their values.

[00:42:50] swyx: Yeah. This is at the absolute limit of my physics understanding, but one example that comes to mind is basically having to solve like ba the equivalent of a three [00:43:00] body problem in a deterministic Well, where the video models, which is approximated good enough. Yeah.

[00:43:08] Right. Like there’s, there’s some point at which your approach kind of runs into like the you now have to simulate the world. Please, thank you very much. And like you’re trying to do that, but only to the extent that the game engine lets you and like game engines cannot do some things.

[00:43:23] Fan-yun Sun: Yeah, no, I mean, I think the interesting or more technical question here actually is where do you draw the boundary between.

[00:43:32] What’s handled with, let’s say, diffusion prior and what, when? What’s handled with symbolic priors?

[00:43:38] swyx: Yes.

[00:43:38] Fan-yun Sun: Okay.

[00:43:38] swyx: Okay.

[00:43:39] Fan-yun Sun: Right. Let’s go there. Because this, this boundary can actually be fluid. Like I think like maybe what you’re trying to get at is like, okay, people are saying pixel prior, everything. But what we’re saying is, okay, there’s a boundary that we draw where this is where we think provides the most economical value for the domains and things that we care about today.

[00:43:59] [00:44:00] And I actually do think, and it’s something that we do internally all the time, which is like, okay, given new equations that we learn or new elements of the world and that we, we learn, or maybe some other knowledge that we acquire in the process of developing the models. Should we still be maintaining this line exactly as it is today?

[00:44:22] Or should we move it a little bit left or a little bit right? Right. Like sometimes that we realize that, oh, like maybe customers or, or folks like want certain things that are better handled with preop pryor as opposed to, symbolic prior than,

[00:44:34] swyx: yeah. Your, your skin thing is a, is a example moving it, right.

[00:44:37] Yeah.

[00:44:37] Or left. Yeah,

[00:44:37] Fan-yun Sun: exactly.

[00:44:38] swyx: I dunno what the, the left right is.

[00:44:39] Fan-yun Sun: Yeah, yeah, yeah. No the, the model.

[00:44:42] swyx: Yes.

[00:44:42] Fan-yun Sun: Actually we have a few iterations of them. They’re actually at slightly different

[00:44:45] swyx: I know boundaries. You should, you should do that. That’s a cool dimension to show.

[00:44:49] Fan-yun Sun: Yeah.

[00:44:50] swyx: Is quantum mechanics the diffusion prior of our world?

[00:44:55] Right. It’s like that’s the boundary of classical mechanics versus quantum. Right? Like, that’s it. At one [00:45:00] point God plays dice and the other point doesn’t.

[00:45:02] Fan-yun Sun: I dunno if Chris, you wanna say it, but I think, I think generally I feel like physics is better with symbol P priors.

[00:45:08] Chris Manning: Even quantum physics.

[00:45:09] Fan-yun Sun: Even quantum physics.

[00:45:11] swyx: Yeah. This is starts against to, MLST territory is, is what I call it, where, he, he likes to get philosophical. We, we we’re quite friendly.

[00:45:18] Vibhu: I mean, we need to get, we need to get singularity. I heard some of that.

[00:45:23] swyx: No, no, I think that is actually really helpful and man, I just want you to productize this like, as a product guy, I’m just like, oh, also

[00:45:32] Vibhu: a gamer, I

[00:45:33] swyx: wanna, it’s like a researcher, like, it’s cool.

[00:45:35] Like this is a, the theoretical, like you have a very good, I don’t know, like the way of thinking about these things, but I just wanna see you like, express it. I do think like your fundamentally things when, when you leave open new tools, like, okay, use, use human intent to incorporate it into how you render.

[00:45:52] Artists are gonna have to take like two to three years to figure out what to do with this. And you just don’t know.

[00:45:57] Chris Manning: Right. But I think, this is, [00:46:00] gives a much more approachable and controllable world for the society, which is the beauty, the beauty of, NLP, that that will enable it to be adopted and used.

[00:46:10] And we are very hopeful about that. Yeah,

[00:46:13] Fan-yun Sun: yeah. Yeah. I mean, we are, we are very focused actually on commercialization in the sense that like we do, we do really believe in the data flywheel app approach. Yeah. Where, we put this in the hands of the creators and the users and then they will teach us when, what capability our model should improve.

[00:46:27] And that’s why we are, we are actually, like products and beta

[00:46:31] swyx: Yeah. Focusing on gaming. What, what’s like the adjacent thing to gaming

[00:46:34] Fan-yun Sun: embody adjacent, basically. So maybe we can, we can I’ll maybe start with where we see the platform in three years. Yeah. Which is like, okay. The users would tell us what they want to achieve.

[00:46:45] The end goal could be, Hey, I just, I wanna make something to teach my kids the value of humility. Or it could be, Hey, I wanna fine tune my, drones to be really good at rescue situations. I could be vacuum robots. I want to like train [00:47:00] my manipulation or like vacuum robot to be very robust to my office, right?

[00:47:04] But it’s like, whatever it is, scenario robust to

[00:47:06] swyx: my office

[00:47:07] Fan-yun Sun: or like navigate very robustly in my office. But then it’s like, whatever end goal that you want, our role model will say, okay, given what you want to achieve, let me generate a distribution of environments such that I can train and evaluate whatever it is you want.

[00:47:24] Yeah. Right. Maybe for the purpose of games, it’s just the end simulation and that’s the end product for certain policies. It’s like I can train it within these environments and then help you see where your policy is failing or not. Yeah. And then, so I think,

[00:47:37] swyx: so in that case, much more of a training tool.

[00:47:40] Than in other training

[00:47:41] Vibhu: evaluation? Both. Right?

[00:47:43] swyx: Sure. Same. Same thing.

[00:47:43] Fan-yun Sun: Yeah, same thing. I think it’s just this role model that allows people to train any policy that can act in any multimodal environments.

[00:47:51] swyx: Would it be harder to reward hack? Is there an angle here where it is harder to reward hack? Like it’s just, I’ll just put it generally because I think that’s a, that’s obviously a key [00:48:00] problem that a lot of people face when in training agents in these environments, and I don’t know, can you solve it?

[00:48:07] Chris Manning: I think not necessarily. To the extent that there’s a mis specified reward that. It seems like it could be hacked in a more symbolic world or in a more pixel based world. I dunno if Sun’s got any thoughts, but I don’t think that’s really being solved.

[00:48:26] swyx: The other thing that comes to mind is just you could just build a better sawa as a video generator model, right?

[00:48:31] Because then you, you would move the diffusion, side a bit more further to the right. I think if I got the directionality correct. And that’s it.

[00:48:40] Vibhu: It’s better on domains, right? Like on consistency over now, or for sure it exists versus something doesn’t, right.

[00:48:46] Chris Manning: So

[00:48:46] swyx: yeah. Yeah. Is

[00:48:49] Vibhu: is a question more like, like

[00:48:51] swyx: I’m just riffing on like, how do you, what can you build, you know?

[00:48:54] Oh, with the stuff that you have. I do think that the minor, the academic does go immediately to training [00:49:00] and in eval evaluation, but like art tends to take unusual directions. Like you might end up,

[00:49:06] Chris Manning: okay. Yeah. But the question is, can you use this piece of software to develop compelling gameplay and. I don’t think you can take SOAR and produce compelling gameplay, right?

[00:49:19] If you want to have a world that you can wander around in a bit, you are good. But what are your abilities to have gameplay mechanics implemented the way you’d like them to be and to have things stay, with the long-term history of your gameplay that influences future actions. I think there’s just nothing there for that.

[00:49:39] swyx: Yeah, I do tend to agree. I, I’m just trying to sort of test the boundaries. I would also make the observation that as AAA games industry has developed the line between what is a movie and what is a game has blurred. And you, you, you do end up basically producing a two hour movie as part of your game.

[00:49:57] Fan-yun Sun: No, honestly, there, there’s so many actually [00:50:00] applications in adjacent markets that our world model can go into. Yeah. But yeah, it, it’s sort of fun to riff, riff on. Although on the execution side, we we, we need to stay focused with like, okay, what are the capabilities we want to unlock over time?

[00:50:11] And there’s a roadmap for that. But yeah, if we’re just riffing on sort of like the possibilities, I feel like, whether it’s endless Yeah, it’s like classic

[00:50:18] swyx: and the embedding for a possibility and endless in my mind, it’s very close. Yeah. I do wanna, focus on one, like weird choice. I, I don’t know if it’s weird.

[00:50:28] Maybe I’m, I got something here. Audio, right? You could have just said no audio And audio in my mind has a lot of recursion, whereas in video you can just do recasting and that’s much computationally much simpler. Audio just seems way harder. I don’t know if you wanna just comment on just the special 3D audio.

[00:50:46] Problem. Did you really have to do it? I guess you do to be immersive, but like a lot of people do treat it as like, well, you just stick a, a tt S model on top of

[00:50:57] Vibhu: Well, there’s a lot more to game audio than [00:51:00] just speech. Right. It’s not just

[00:51:01] swyx: tts. Yeah. Tts. S Fxt, GM Spatial in my mind Echoes

[00:51:06] Chris Manning: Yeah.

[00:51:06] swyx: And reflections.

[00:51:07] And I, I don’t even know what’s, what else? I don’t know what, what other problems in this space.

[00:51:13] Fan-yun Sun: Yeah, I think this point like the, it’s sort of a more, more pointing to the benefits of using an game engine as a tool that’s available to the model, right? Because like part of the spatial audio is from the code that is underlying the simulation.

[00:51:32] And while we do give our model access to other types of audio models as. Tools.

[00:51:39] swyx: None of them would be spatial, I think.

[00:51:41] Fan-yun Sun: But that’s exactly sort of more 0.2. We’re giving our model an abstraction or a suite of tools such that it’s able to achieve that. And you can argue that sort of spatial is like a, like a emergence out of the, the tools that we and abstraction that we provide to the agents.

[00:51:59] And I think that’s the beauty of [00:52:00] this, this, this approach is like there’s a lot of things kind of like how human’s built technology and they’re like Lego blocks that build on top of each other. And it’s the same thing here. There’s gonna be things that sort of just sort of emerges from being able to put these things together in like combinatorially interesting ways,

[00:52:14] Chris Manning: right?

[00:52:15] So this integrated audio model exploits the understanding and semantics of the Moon Lake world, right? And whereas in general for the Gen AI video models. There’s no actual integration across to audio at all, right? That someone might stick some music or stick a soundscape or whatever else on top of their video.

[00:52:44] So it’s not a silent video, but they’re in no way connected into a consistent world model. And there’s nothing that’s okay. An action is happening in the video. Therefore there should be a sound that’s [00:53:00] coming from this part of the visual field.

[00:53:03] swyx: Yeah.

[00:53:03] Vibhu: Is that different than Sora too? Does it not have audio?

[00:53:06] Not to say it’s not like

[00:53:08] swyx: amazing

[00:53:08] Vibhu: isn’t a spatial

[00:53:09] swyx: audio.

[00:53:09] Vibhu: It doesn’t,

[00:53:10] swyx: no. I’ve played around it with it enough. It just sounds like someone put an 11 laps voice on top of it and just tried to do the lip sync.

[00:53:18] Vibhu: Oh, yeah. I’ve seen, okay. Generate a dog at the beach and reactions to big wave and move

[00:53:23] swyx: around.

[00:53:23] It’s definitely like, so have the dog, have the dog move away from camera and see if the, the song goes down. It doesn’t. ‘Cause they don’t have facial audio.

[00:53:32] Fan-yun Sun: We do want to basically like we, our moral model, like the one we’re training is basically towards the goal of having a combined latent representation across all these different modalities.

[00:53:42] Right? Such that it can like reason across these different modalities. So for example, if I close my eyes and like you play a video, you play a sound of like a car skidding away from me. I almost can like, visually extrapolate that trajectory in my mind. And I think that type of capability, we want our model to be able to reason, right?

[00:53:59] And that’s the reason that [00:54:00] we’re sort of taking this multimodal reasoning approach. It’s like we want this combine late in space that can

[00:54:05] swyx: Yeah. Oh, you said late in space. We like that. Here we have to play the, the bell Every time that someone says late in space, no, you gotta train daredevil one. Where you, you, you, it’s only audio, but you have to work out.

[00:54:15] Where everything is.

[00:54:19] Cool. I I think that that was, that was about it for our Moon Lake coverage. I do think that we have like a couple of, Chris Madden questions on, on IR and, just any, any other sort of attention topics or n NLP topics.

[00:54:31] Vibhu: Okay.

[00:54:31] swyx: Go ahead.

[00:54:32] Chris Manning’s Journey: From NLP to World Models

[00:54:32] Vibhu: Well, no, I mean, yeah, it’s just fun. We talked a bit about how you guys met, but you basically, you, you were like the godfather of NLP per se, right?

[00:54:39] You spent the whole career from early embeddings, early early attention. You did 2015 attention for machine translation, everything. You, you had information retrieval, so RAG before rag, we just wanna shout that out and admire a lot of that. Right? So what prompted the switch over to world models?

[00:54:56] How, how’d all that come about?

[00:54:58] Chris Manning: To some answer it [00:55:00] is, the enthusiasms and creativity of students, but there’s a bit of a history there, right? So, yeah. So clearly most of my career has been doing stuff with language and how I got into research was thinking, ah, this is just so amazing how humans can produce speech and understand each other in real time.

[00:55:21] And somehow they managed to learn languages from their kids. How could this possibly happen? And so, yeah, starting off I was very focused on language, but as it sort of got into the 2000 and tens, I started, going, I’d been working on question answering, and then I started to get, interest in visual question answering.

[00:55:42] And that was an area where it was very noticeable. That the visual understanding was bad. Right. These were the days when like, it sort of seemed like there’s almost no visual [00:56:00] understanding. You were just getting answers that came from priors. So, if you asked how many people are sitting at the table, it’d always answer two regardless of how many, how many people you could see in the picture.

[00:56:11] And so it seemed like, oh, these models actually aren’t able to get semantic information outta IMA images. And so I was interested in that problem and tried to work more on that. And so then that required. Knowing more about what’s happening in vision and how you can represent visual information.

[00:56:34] And then things start, there started to be this revolution of, doing generative AI images. And then I had students that started looking at that before the era of Moon Lake. I was also working with Demi Gore, who founded pika. And so, and

[00:56:50] swyx: Ian obviously

[00:56:52] Chris Manning: with gans. Yeah. Though Ian was never my student, but yeah, Ian I was very aware for the, the whole decade there of Ian with Gans.

[00:56:59] [00:57:00] Yeah. And I mean, Ian was a Stanford undergrad, but yeah,

[00:57:03] Vibhu: richard des u.com, I believe he was your student.

[00:57:06] Chris Manning: Yeah. Yeah. And there were, there were links across at that stage as well. So there were several papers in that era of doing, I mean, so Andre Cap was a, PhD student at the same time as Richard.

[00:57:20] And so there was some joint language vision work in that era as well. It seems kind of ancient by modern standards, but yeah, we’re trying to go from sort of textural dependency graphs to visual scenes

[00:57:32] Vibhu: at a time. The glove embeddings really took over a lot of. T-F-I-D-F, like one hot encoding, all that.

[00:57:38] The early vision language models we saw were like lava style adapters, right? It’s, it’s technically still just embedding latent space. Let’s add image, let’s like mixed modality. So, and that, that’s one of the things you super put out there too, right?

[00:57:51] swyx: Yeah.

[00:57:51] Vibhu: Yeah.

[00:57:52] swyx: Yeah.

[00:57:52] Hiring, Closing & The Name “Moon Lake”

[00:57:55] swyx: Well, thank you for all of that. Thank you for all advancing the worlds on, world modeling.

[00:57:56] I honestly, do think that if people deeply understand everything we just [00:58:00] covered, they will see what’s coming. I think you guys have, made some, a really significant contribution here. What are you hiring for? What is the, what do people find? We, we agreed that the CTA was a hiring call.

[00:58:10] Yeah. Don’t we have a GI You don’t need, you don’t need engineers anymore, right?

[00:58:14] Fan-yun Sun: Yeah. On the model side we are actually striving towards basically a self-improving system. But what that means is that we need people to set up the self-improving system. So more, more specifically people who have the intersection of knowledge within co-generation and computer vision and graphics, right?

[00:58:30] Yeah. That’s, that’s sort of the core research background that we look for within OTM and, and the majority of the team today do have like both backgrounds.

[00:58:38] swyx: When you say computer vision and graphics, are they the same thing or is it computer vision one thing, graphics, another thing. And how intertwined are they?

[00:58:46] Chris Manning: They’re intertwined but different.

[00:58:49] swyx: Yeah.

[00:58:49] Chris Manning: And I think, this relates to some of the themes that we’ve been talking about, that the more explicit underlying [00:59:00] world models that are being constructed inside Moon Lake really draw on the computer graphics tradition. And so it’s then combining that with the visual understanding of vision.

[00:59:16] swyx: Got it. Yeah. All right. So you’ve written a game engine, you’re come talk to us, right?

[00:59:21] Fan-yun Sun: Oh yeah, definitely. Definitely. But I do think that the line is blurred, like increasingly blurred these days where it’s like if you have a general understanding of group vision and graphics,

[00:59:31] swyx: I think for your standards it is, for me it feels like vision is, is.

[00:59:35] I’ll leave that to the big labs graphics. I, I, I can get that, you would want to do that from more first principles, but vision, there’s so many vision models off the shelf that I can take, but probably not good enough for your

[00:59:45] Fan-yun Sun: I see, I see. If, if you’re sort of like making that distinction then maybe we, we care a little bit more about having graphics

[00:59:51] swyx: knowledge.

[00:59:51] Yeah, exactly.

[00:59:52] It could be like, sometimes a hiring call can be as simple as like, if you know the answer to blah, you should talk to me. Like the sort of core known hard [01:00:00] problem in, in your world.

[01:00:01] Fan-yun Sun: Ah, I see. Yeah. In that case, if you, yeah, definitely. If you’ve written a game engine before, if you’ve rld a variety of coding models on different objectives, like

[01:00:13] swyx: easy,

[01:00:13] Many of those, yeah.

[01:00:14] Fan-yun Sun: If you’ve done multimodal lean space alignment, I, I intentionally include

[01:00:20] swyx: space.

[01:00:20] Fan-yun Sun: Again,

[01:00:21] swyx: a poor editor has a thing every time. Yeah. Lean space alignment. Honestly. Is it that hard?

[01:00:26] I, I, there’s some scripts out there that I’ve saved for the day. I someday have to do it, but I don’t have to do it.

[01:00:31] But it’s

[01:00:32] Fan-yun Sun: done, I think. Yeah. There, there’s, there’s a versions of that that are done. But I, I think we are aligning audio, text, language and video. Yeah. Right. Like, and basically we have these role models that are able to act as agents to like act in these worlds and extract long horizon videos and encoding that back to the model to sort of self-improve.

[01:00:52] So it’s an insanely exciting, but also technically challenge problem. Yeah. So people who wanna do their lives best work, that only [01:01:00] makes a place.

[01:01:01] Vibhu: How big are you guys? Where are you guys based?

[01:01:02] Fan-yun Sun: We’re currently based in San Mateo, although we’re moving up to sf. We’re about 18 folks right now.

[01:01:08] swyx: My ending question was gonna be why, what, what is the name?

[01:01:10] What’s behind the name?

[01:01:11] Vibhu: Yeah.

[01:01:12] Fan-yun Sun: Oh,

[01:01:14] Vibhu: Very cool. Graphics and design, by the way.

[01:01:16] Fan-yun Sun: Actually at the, at the time when the, when the, when we started the company, we were thinking a lot about how do we make a company name that gives people the vibe of like, open ai, but for like, almost like industrial light and magic vibes.

[01:01:28] Wow. Because it’s like we care about creativity and using that as a funnel to solve a GI. So then we were, we, we brainstorm a lot around like Dreamworks, right? Like industrial light magic. And, so there’s a few, few basically, space of things that we feel like are very, very semantically close to the company’s identity.

[01:01:47] swyx: Yeah.

[01:01:48] Fan-yun Sun: And then it ended up being Moon Lake, partly because of the Dreamworks vibe, the Dreamworks, moon

[01:01:54] swyx: Lake.

[01:01:55] Fan-yun Sun: Exactly. Yep. So that was a little bit of that inspiration. And then the moon was sort of [01:02:00] like a, it basically was like about the. Reflection. The reflection part also implies the self-improvement loop.

[01:02:07] Wow. That we sort of like, that’s really bleed and that’s the path towards multimodal general intelligence. So that’s, that’s that. I’ll leave that as I love a good

[01:02:15] swyx: name. I love a good name. This is great. It’s a

[01:02:16] Vibhu: very

[01:02:17] swyx: good name. It’s very good. Lo I’m glad I asked the question. I will also say, one, my favorite story, books or biographies ever is, creativity Inc.

[01:02:24] With Ed Kamal’s, story about Pixar and how he, was rejected as a Disney animation artist. So then he went into computing and brute forced his way into back. No, I love that story. Yeah. Disney.

[01:02:37] Fan-yun Sun: Yeah. And Walt Disney is also like one of my favorite founders. He’s like, his, his story. Like at the time you’re like, okay, I’m gonna create this like.

[01:02:44] Immersive park. Like people can’t, don’t even have that technology to create it virtually, but they’re like, you know what, let’s just build it physically such that people can,

[01:02:50] swyx: so he is the first world modeler.

[01:02:52] Fan-yun Sun: No, I, I I tell people that like, theme parks are world models too.

[01:02:56] swyx: Mm. Yeah. Yeah. Yeah. I mean, it’s a small world or it’s [01:03:00] a, like the Epcot center with all the little, replicas of the countries.

[01:03:03] Yeah. Those are very interesting. Okay. Well thank you, we’ve covered, a huge amount. Thank you for your time and thank you for inspiring us.

[01:03:10] Fan-yun Sun: Thank you

[01:03:10] swyx: for having us. Thank you. It’s fun

[01:03:11] Fan-yun Sun: chatting. Yeah. It’s been a good time.

Mistral has been on an absolute tear - with frequent successful model launches it is easy to forget that they raised the largest European AI round in history last year. We were long overdue for a Mistral episode, and we were very fortunate to work with Sophia and Howard to catch up with Pavan (Voxtral lead) and Guillaume (Chief Scientist, Co-founder) on the occasion of this week’s Voxtral TTS launch:

Mistral can’t directly say it, but the benchmarks do imply, that this is basically an open-weights ElevenLabs-level TTS model (Technically, it is a 4B Ministral based multilingual low-latency TTS open weights model that has a 68.4% win rate vs ElevenLabs Flash v2.5). The contributions are not just in the open weights but also in open research: We also spend a decent amount of the pod talking about their architecture that combines auto-regressive generation of semantic speech tokens with flow-matching for acoustic tokens (typically only applied in the Image Generation space, as seen in the Flow Matching NeurIPS workshop from the principal authors that we reference in the pod).

You can catch up on the paper here and the full episode is live on youtube!

Timestamps

00:00 Welcome and Guests00:22 Announcing Voxtral TTS01:41 Architecture and Codec02:53 Understanding vs Generation05:39 Flow Matching for Audio07:27 Real Time Voice Agents13:40 Efficiency and Model Strategy14:53 Voice Agents Vision17:56 Enterprise Deployment and Privacy23:39 Fine Tuning and Personalization25:22 Enterprise Voice Personalization26:09 Long-Form Speech Models26:58 Real-Time Encoder Advances27:45 Scaling Context for TTS28:53 What Makes Small Models30:37 Merging Modalities Tradeoffs33:05 Open Source Mission35:51 Lean and Formal Proofs38:40 Reasoning Transfer and Agents40:25 Next Frontiers in Training42:20 Hiring and AI for Science44:19 Forward Deployed Engineering46:22 Customer Feedback Loop48:29 Wrap Up and Thanks

Transcript

swyx: Okay, welcome to Latent Space. We’re here in the studio with our gues co-host Vibh u. Welcome. Thanks. Excited for this one as well as Guillaume and Pavan from Mistral. Welcome. Excited to be here.

Guillaume: Thank you.

swyx: Pavan, you are leading audio research at Mistral and Guillaume, you're Chief Scientist,

Announcing Voxtral TTS

swyx

Host

(00:05) Okay. (00:05) Welcome to Lean Space. (00:06) We’re here in the studio with trustee co-hosts, Vibhu. (00:09) Welcome.

Vibhu

Host

(00:11) Very excited for this one.

swyx

Host

(00:12) As well as Guillaume and Pavan from Mistral. (00:15) Welcome. (00:16) Excited to be here. (00:17) Thank you for having us.

(00:18) Pavan, you are leading audio research at Mistral and Guillaume, you’re a chief scientist. (00:23) What are we announcing today where we’re coordinating this release with you guys?

Guillaume

Guest

(00:26) Yeah, so we are releasing Voxtral TTS. So it’s our first audio model that generates speech. It’s not our first audio model. We had a couple of releases before.

(00:35) We had one in the summer that was Voxtral, our first audio model, but it was like a transcription model, ASR. Like a few months later, we released some update on top of this, supporting more languages. Also a lot of table stack features for our customers, context biasing, precision, timestamping and transcription. We also have some real-time model that can transcribe not just at the end of the level.

(00:56) You don’t need to fill your entire audio file, but that can also come in real-time. And here, this is a natural extension in the audio, so basically speech generation. So yeah, so we support nine languages, and this is a pretty small model, 3D model, so very fast, and also state of the art. Performed at the same level as the base model, but it’s much more efficient in terms of cost, and also much, in terms of cost, it’s also much cheaper, only a fraction of the cost of our competitors.

(01:22) And we are also releasing the work that this model is running.

swyx What’s the decision factor?

Guillaume It’s a good question.

swyx

There will be more. Yeah, Pavan, any sort of research notes to add on?

Architecture and Codec

Pavan: But it’s a novel architecture that we develop inhouse.

We traded on several internal architectures and ended up with a auto aggressive flow matching architecture. And also have a new in-house neural audio codec. Which, converts this audio into all point by herds latent [00:02:00] tokens, semantic and acoustic tokens. And yeah, that’s that’s their new part about this model and we’re pretty excited that it’s, it came out with such good quality and Jim was mentioning. Yeah, it’s a three B model. It’s based off of the TAL model that we actually released just a few months back and insert trunk and mainly meant for like the TTS stuff, but they need text capabilities are also there. Yeah.

swyx: So there’s a lot to cover.

I always I love any, anything to do with novel encodings and all those things because I think that’s obviously I creates a lot of efficiency, but also maybe bugs that sometimes happen. You were previously a Gemini and you worked on post training for language models, and maybe a lot of people will have less experience with audio models just in general compared to pure language.

What did you find that you have to revisit from scratch as you joined this trial and started doing this? At least

Understanding vs Generation

Pavan: when it comes to, for, I think the, there are two buckets, I guess the audio understanding and audio [00:03:00] generation. The audio understanding, like the walkthrough models that Kim was mentioning that we released earlier.

The walkthrough chat that we released I think July last year, and the follow up transcription only, models family that we released in January, that would be one bucket, and the generation is another bucket. I think. You can also treat them as a unified set of models, but currently the approaches are a little different between these two.

To your question on how audio is fed to the model? In the understanding model, it’s very similar to actually Pixar models that we also released,

swyx: yes.

Pavan: That’s

swyx: amazing.

Pavan: It was pretty, I, that was the first project I worked on after joined Misra. It was pretty, pretty nice. And Wtu was very similar in spirit.

I guess So we feed audio through an audio encoder similar to images through a vision encoder, and it produces continuous embeddings and which are fed as tokens to the main transformer decoded transformer model. Yeah. On the model output is just text. So on the output side, there is nothing that needs to be done in these kinds of mode.

I [00:04:00] guess the interesting part of what the generation stuff is, the output now has to produce audio and. The approach that we have is this neural audio codec, which converts audio into these latent tokens. There is a lot of existing attrition and a lot of models which are based off of this kind of approach.

And we took a slightly. A different, design decisions around this. But at the end of the day, the neural audio product converts audio into a 12.5 herdz set of latents. And each latent is, has a semantic token and a set of acoustic tokens. And the idea is that you take these discrete tokens and then feed it on the input side.

There’s several ways to use this at each frame, but we just sum the embedding. So it’s like having key different vocabularies. Combine all of them because they all correspond to one audio frame on the input side. The output side is the interesting part on the output side, the, it’s not the, I don’t know if it’s the most popular, but one.

Popular technique is to have a depth transformer [00:05:00] because you have K tokens at each time step, like with a text, you just have one token at each time step. So you just do predict the token from the vocabulary with, yeah, with just, you get probability

swyx: This’s a very straightforward text. Very

Pavan: straightforward.

swyx: Yeah.

Pavan: But if you have K tokens, then the name thing would be to predict all of them in paddle. That doesn’t work. At least that doesn’t work that well because audio has more entropy. And the, one of the techniques people use is this depth transformer where you you almost have a small transformer, or it can be L-S-T-M-R in as well, but people use transformers and you predict the K tokens in auto aggressive fashion in that.

So you have two auto reive things going on.

Flow Matching for Audio

Pavan: So the thing we did differently is in, instead of having this auto aggressive K step prediction, we have a flow matching model. Instead of modeling this as a discrete token set we trained the codec to be both discrete and continuous to have this flexibility.

So we did try the discrete stuff too, and which it works well, but the continuous stuff works just better. So yeah, we took this flow matching, so the, it’s a flow [00:06:00] matching head, which takes the latent from the main transformer and like kind in fusion, it’s denoising, but in this flow matching itself, velocity estimate.

So you go from this noise t all the way to there. Audio latent, which corresponds to the 80 millisecond audio and then, which is sent through the work order to get back the 80 millisecond audio frame.

swyx: Yeah. Is this the first application of flow matching in audio? Because usually I come across this in the image.

Pavan: Yeah. Actually, in some sense there are models flow matching models in audio, but I think this specific combination I could be wrong. There could be somewhat. No. I haven’t seen. I haven’t seen much work in this, so I think it’s novel and a lot of it’s just a way bigger community, so they, I think they pioneer a lot of these diffusion flow matching work, and it’s interesting to adopt some of the ideas there into audio and,

swyx: yeah.

Pavan: Yeah, I’m, personally that’s the think part which is trying out about. One of more meta point is unlike text, even in vision, I think this is true, but in [00:07:00] audio step literature that there is no.

Winner model, yet there is no, okay, this is the way you do things. It’s it’s still by, I think people are still iterating and figuring out like what’s the best overall recipe. I guess the idea. Pretty sure there are models which are also completely end-to-end, like NATO audio. NATO audio, but it’s still not come to a convergence point where this, the right way to think that.

That also makes. A space pretty exciting to explore.

Real Time Voice Agents

Vibhu: What are some of the ways to look at it?

Vibhu: There are ways where you can do diffusion for audio generation, but if you want like real time generation, that’s a big thing with the approach I’m assuming that you took. Yeah. And also like how do you go about evaluating different axes of what you care about, yeah,

Pavan: good point. I think we so you can do just flow matching diffusion for the whole audio. We didn’t even go down that path because one of the main applications is voice agents and we want real time streaming, and that’s the use case. That’s not the only use case, but that’s one of the primary use cases we want to get to.

So we [00:08:00] picked the auto aggressive approach for that. And within the auto aggressive space, again, you can do chunk by chunk or you can do so we picked the. I think at least personally prefer the operations, which are the simplest, and so we try to see, can we just add audio as just another head to our regular transformer decode model because that kind of makes it easier for eventual end-to-end modeling of audio text native modeling.

Yeah. And it works pretty well. So I guess we went with that and we tried a little bit, but the flow matching head itself, like we had a discreet. Diffusion kind of approach, which also works well, but the flow matching work better.

swyx: I was just curious about how you also think about this overall direction of research.

Do you basically, when you work with the audio team, do you set some high level parameters and then let them explore whatever, or how does it work between you guys?

Guillaume: No I think the way it works is that we are the, we are prioritizing together, I think, what are the most important features because there are many things we can do [00:09:00] in audio.

Yeah, I think we try to. These are like how we should do things, for instance. Ultimately what we want to do is to build this through duplex model, but we are not going to start this start there directly, I think is. Some of the project people are doing, but

swyx: just to confirm, full effects means it can speak while I’m speaking or,

Guillaume: yeah.

Okay. Audio. Yeah. Yeah. So intimately we’re going to get there, but for us it was, we decided to take it like a step by step. So we start with whatever is the most important. I think support customers, which is the transcription is the most popular use case. Then the speech generation, Soviet time, just a bit before that.

And then actually to be like more, but try combining everything all together. But but yeah, we thought it was also important to like separate things and optimize each capability one by one before we

swyx: measure of that together. And the super omni model. But

Guillaume: very interesting because as Par said, it’s when you work on some other domains of this airline and everything, there are many areas where I think it’s not as interesting.

For instance. Many places, it’s essentially just around data or like creating new environments on a lot of kind [00:10:00] of easy things. But things were, I think the research is maybe not as interesting. Were in audio. There are so many ways to actually build this model. So many ways to go around it. That’s the sense I think is really interesting.

And what we also tried for speed generation is that we tried multiple approaches. What was interesting that even though they were extremely different, they under the big know the particles but the for matching turned out to be quite more natural. So we are happy with this.

swyx: Is there intuition why it maybe like flow matching is just models speech better in some natural fundamental, latent dimension?

Pavan: No, I think the main thing is e even at a particular time step, there is a distribution of things.

swyx: Yes.

Pavan: To be predicted like the way you inflate. So you already know the word that you’re speaking and Yeah. The intake space, let’s say the word maps register a single token for simplicity.

In most cases it does. So there is not a lot of so you just pick the word, but with within audio, even the same word could, even with your own voice, could be inflicted in so many different ways. And I think [00:11:00] any approach which like models this distribution and. And flow matching is one, one of the take.

It’s not the only one at all, but it’s a one which works pretty reasonably well. I think that’s better. So you have to pick across several different, the intuition I have is it’s, there are some, several different clusters each corresponding to some specific way you would inflict, pronounce that thing.

And you can’t predict the mean of it because that corresponds to some blurred out speech or something like that. But you have to pick one. And then like sharp

swyx: conditional inference.

Pavan: Yeah, exactly.

swyx: Is that all covered under disfluencies, which is I think the normal term of art. Pauses intonations. By the way, I have to thank Sophia for setting all this up, including like some of these really good notes because

Pavan: Yeah.

swyx: I’m less familiar with the audios for me.

Pavan: No. I think dis dismisses are definitely one such Eno defenses is more like

swyx: which is arms are.

Pavan: Yeah, arms. And also repeat like you like,

swyx: yeah.

Pavan: You do this full of words, your thinking, so you repeat the word.

swyx: Okay. Whereas intonation is like a diff, it’s up up [00:12:00] speak and all this.

Okay.

Pavan: Yeah. So I think there is a lot of like entropy. And modeling it as a distribution. And a, any technique which helps with it and the depth transformer is a conditional way of modeling this. And Transformers actually really good at it, even though that’s a mini transformers. So I think that worked pretty well too for us too.

It’s just that the main concentration is when you have a depth transformer. If you have K tokens, you need to do K auto steps, right? Even though it’s a small thing, it’s K steps, which is very vacant, say heavy, but flow matching. We were able to cut it down significantly. So we are able to do the inference in quad steps or 16 steps and it works pretty well.

And there are more normal techniques to bring it down even further to like, in extreme case, one step like we’re not doing it yet, but it at least the framework, LEDs itself to more efficient and Yes.

swyx: And the image guys have done.

Pavan: Yeah.

swyx: Incredible work guys. Yeah.

Pavan: It now you just. Send a prompt and you get an image.

swyx: Yeah. Surprisingly not enough. I think image model labs use those techniques in production. I think it’s, I feel like it’s a lot of research demos, but [00:13:00] nothing I can use on my phone today.

Guillaume: The thing, there’s a thing that would be interesting here is that since, indeed I’ve been so much sure that has been done in the vision community compared to radio dys, stomach, I think there are so many long infra Yeah.

And there are so many things we can do to actually improve this further. So it’s our first version, but we have so many ways to exist, much better and much more efficient, cost efficient, so

swyx: yeah.

Guillaume: So really it’s not a new field at all, of course, but there are still so many things that can be done.

Perfect. It’s

swyx: nice. I should also mention for those who are newer to flow matching, I think the creator, this guy’s name is Alex, he’s done I think in Europe’s maybe two Europes as ago. There was, there’s a very good workshop. There’s one hour on like this matching is I would recommend people look that up.

That’s the other thing, right?

Efficiency and Model Strategy

swyx: The efficiency wise, like I, I imagine like the reason is open weights the reason you pick 3.6 B backbone it you are 3.4 B you are, try to fit to some kinda hardware constraints. You kinda fits some kinda basic constraints. What are they?

Guillaume: Not necessarily, I think something we care about in our model that they’re efficient.

So we have a [00:14:00] lot of separate model, for instance. So we have this that is very small, very efficient. We also have a small OCR model that is available. Good, highly efficient as well. And I think on a project maybe there, I think companies are going to take is to have a coverage general model that will do a bit of everything.

But that is also going to be expensive. On here. What want say is if you care about this specific use case, if you can actually use this model, it just does that. It’s extremely good at it. Survey, very efficient. That’s why we can actually add. We do, but also OCR that are like really good at that.

And that would be much more cost effective factors and the general model that will contain a lot of capabilities you don’t really need. So yeah. So we’re doing like general model, but also like more customized model. This,

Open Weights and Benchmarks

Vibhu: how does it compare to other TTS models? It’s, we are going follow open wave.

We’re just dropping it. I think it’s pretty good.

Pavan: Yeah, I think it’s pretty good. Like it, it’s definitely one of the best. For sure. It’s probably I would say it’s the best open source model, but

Vibhu: decipher themselves.

swyx: Yeah.

Voice Agents Vision

Vibhu: Why now? How does it fit into broader ral vision? How do you see voice agents?

How do you see voice? I think every year I’ve heard, okay, you’re a [00:15:00] voice. You’re a voice. There’s a lot of architectural stuff. There’s a lot of end time that see it, your solving, but where do you see voice setting?

Guillaume: We had so many customers asking for voice. That’s also why we wanted to build it.

What’s interesting in this domain is that. In a sense, if you take something simple like transcription it doesn’t seem like something that should be very hard to do for a model. It’s essentially, it’s pattern recognition. It’s classification on this. Models are very good at classifying, right?

Or nonetheless, when you talk to them it’s not there yet, right? It’s not, you don’t talk to them the same way you talk to a person. On something, maybe people don’t realize it. It’s in English it’s still much better than in any user language, even compared to French instance. If you talk to this million in French, when you see people talking to this they’ll talk very slow.

They’ll articulate as much as they can. So it’s not natural, right? We’re not yet to this. And I think, yeah, maybe the next generation will not know this, but yeah, I think people that. But our edge will actually always keep this bias speaking very slowly when they talk to this model. Even if maybe, probably in a couple of years, maybe next year it’ll not be necessary anymore.

But yeah. But what’s interesting is to see that yeah, even for like languages [00:16:00] like yeah, French and Spanish Germans that are not no, no resource on religion. You have a lot of audios there on still it’s not as good. And I think a consequence. Because then for this, I suppose just is not as much energy, as much effort that has been put done in some other mod that for some vision or like coding.

But but yeah, there’s still a lot of progress to be done. I think it’s just a question of doing the work and it’s clear path I think to get there.

Pavan: It’s a little fascinating because I worked on Google Assistant I think while back at this point, but it’s, I think it’s, it like when you take a step back, it’s fascinating.

It’s not that long ago. It was like four years ago or five years ago, and it’s now it’s completely audio in, audio out and the function calling and the whole thing happens completely end to end. And in a very natural,

swyx: yeah,

Pavan: natural way and still ways to go. Kim was telling, even despite all the previous, it’s not like you’re speaking to a person.

When you talk to any of these agents, bots, or voice mode kind of situation, it’s still like a gap. I think that’s the great part and I feel like with even the existing [00:17:00] stack, we should be able to get to this very natural speech conversational abilities soon enough I guess.

And we’ll also hope. I get that

Guillaume: on this kind of the next step, right? Because when you talk to these agents, like usually people are just writing to them and sometimes they’ll this very clear, for instance, you are, you want to write code, but you are, you have a very clear idea of how you want the model to implement what you in mind.

But so here you are able to spend a lot of time writing. So it’s not really efficient on audio is really like a natural interface that is just not there yet, but I think it’s just gonna be the place.

Vibhu: How’s it like building, serving, inferencing, like we see a lot about, it’s very easy to take LMS off the shelf, serve them.

Fine tuning, deploying. I know you guys have a whole you have Ford, you have a whole stack of customizing, deploying. Is there a lag in getting that. Like distribution channel. Are you helping? There is. So like prompting, lms, you can have them be concise, verbose, all that.

They’re built on LM backbones, these models. How do you see all that?

Enterprise Deployment and Privacy

Guillaume: Yeah, I think this is a lot of what we’re doing with our own customers. Very [00:18:00] often they come to us, so it’s for different reasons. I think one reason is sometimes they have this lot of privacy concerns.

They have this data that it’s very sensitive. They don’t want data to leave. The companies, they wanted to stay. Inside the company. So we have them deploy model in-house. So either on a, either on premise or on private cloud. So they’re not worried that it’s given to a third party on the there some leakage.

Sometimes they have this kind of many companies have this different, sensitivity of data they have like sometimes channel chat can send it to the cloud has to stay there. So then it creates some kind of heterogeneous workflows where it’s annoying. You cannot send some data to the cloud.

This one you can, so here, when we actually deploy the model for them, they don’t have this consideration. They are like not worried that, this is going to leak. Everything is much easier. So we help them basically do this on the, so it’s one of the very proposition. But but the other is very often, when customers use this off the shelf close model, but very sad is that they are not leveraging, these data that have been collecting for four years or something for decades.

So much data. Sometimes it’s trillions of tokens of [00:19:00] data in a very specific domain. Their domain, which is data that you’ll not find in the public, on the public internet. So data on which, like close model, we actually not have access to one, which that’s going to be really good. So if they’re using like closed source models are basically not benefiting from all these insights.

All these data they have collected three years, they can always give it into the context that in France, but is never as good as if you actually train the modern analysis. So yes, that’s basically what we help them to do. We actually provide them some purchase, basically what we announced at GTC this week.

So we provide them with this, it’s basically like a platform with a lot of tools to actually help them process data. Trained on that. Yeah, it’s actually the same thing that we’re using in the science team. So it’s actually very better tested infrastructure, like a lot of efficient training cut base.

For a quality pre-training like a fine tuning, even doing S-F-T-I-L. So we help them do this using the same tools as what our science team is building is using. So since it’s tools that we’ve been using for two years now, it’s really better tested. It’s really sophisticated.

So it’s the same thing. We are giving to them, giving the company the same thing [00:20:00] that what are same still using internally actually build their own ai and it makes a really big difference. I think sometimes customers. And many in general don’t realize how much better the model becomes when you fine tune it on your own data.

And you can have a, your model is here. You start from there. You have a cross source model, which is sort here, but if you actually fine tune it can actually really go much further than this. And then you have a very big advantage. The model is trained on your entire company knowledge, so it knows everything.

You don’t have to feed like 10 K tokens of contact at every query. So it’s it’s much easier. It’s a bit, I think using a closed source model is really sad because it basically puts. You are not leveraging all this data and you are going to be using the same model as all your old competitors when you’re actually using, everything you have been collected for years, which is really valuable.

So yeah. So we help basically customers do this. We have a lot of solution I mean deployed for engineers that go in the company that basically look at the problem customers are facing to look at what they’re struggling to do what we should do to solve it. So we help them solve them together.

So it’s I think our approach is a bit different, but here. [00:21:00] Some of their companies and competitors, it’s, we don’t just release an endpoint on sale, do some stuff on top of that, or we don’t just give a checkpoint. We really look very closely with customers. We look at the issues they have, we had them solve them.

We really make some tailored solution for the client are facing. Some example are also going to be, sometime we have some customers. They really wanted to have a really good model, really performance on some, like Asian languages on the, if you take some of the shelf models, they can speak it, they can write in this language, but it’s not amazing.

This language would be like maybe zero 1% of the mixture. So it has been included during training, but very little. So what we did here is upgrade. We trained a new model for them, but so this language was 50% of the mix, so it’s much, much stronger. It knows of the dialects, it knows the, so it’s yeah.

So it’s some example of things we can do and it’s really arbitrary, custom. I think you had some of their customers, for instance, they wanted some. They wanted some 3D model that can do audio with a very good function cable. So something you wanted to put in the car in particular, they wanted this to be offline because in a car you don’t necessarily have access to internet.

So [00:22:00] yeah. So here we can actually build the solutions. There is no like model out of the box on this. In the internet you have this very, you have this very general model generalist, like he’s strong model. But for things like this, they always want at specific solutions and on some other reasons.

Sometimes they come to us is because, like they, they experiment with some closed source model. They get some prototype. They’re happy with what they build. They, it works well. They’re happy with the performance, and then they want to go to production and then they analyze. But it’s extremely expensive.

You cannot push this. It’s so then they come back to us on this. They can help us build the same thing as this, but using something much cheaper on here. And here we can sometime be something 10 x cheaper by just functioning a model and it’ll be better OnPrem on their old server and also much cheaper as well.

So yeah,

swyx: that’s the drop pitch right there. Take all the

money.

Vibhu: And outside of that you do, we do put open wave models so people can do this themselves. I feel like not enough people go outta their way.

swyx: They’re not going to, they’re gonna ask them to do it as the expert. I

Guillaume: think initially we didn’t know, [00:23:00] we wanted completely short at the beginning of the company because, I think our study was not exactly the same as what it is today, but what we underestimated initially is the complexity of deploying this model and connecting them to everything to be sure it has access to the company knowledge on the, and it was, yeah, on, we were seeing customers struggling with this, but it was even, that was three years ago and no, things are much more complicated because now you don’t just have, text on SFT on a simple instruction following.

You have reasoning like your agents, you have like tools. You have a multimodal audio, so it’s much more complicated than before. And even back then it was hard for customers. So they really need, have some support and this is why actually providing like always some four D position as well. The process

Fine Tuning and Personalization

swyx: I’m curious is there also voice fine tuning that people do?

Pavan: So in this forge we also have a say unified framework. And the hope is like the er speech to text that we released earlier this year. And even the ER chart that we released last year. And I think a big people, I think there’s a big, rich ecosystem [00:24:00] of people fine tuning whisper, and people want the same thing with w so it’s much stronger than Whisper.

And yeah, the the platform offers that kind of fine tuning yeah, which could be any kind of fine tuning. Like for instance, even sometimes people want to support new languages to this, which are tail languages, which we hope to cover. Certain natively, but if there is a language where you data and you want to frank you, I think this is a good use case.

Or the other use cases, you, it’s the same language, like even English but it’s in a very domain specific way.

swyx: Yeah. Terminology, jargon, medical stuff.

Pavan: Exactly. And also there’s specific acoustic conditions like there’s a lot of noise or the, and. The model will do decently in most conditions, but you can always make it better.

And that those are some of the use cases where you can improve it e even further. And that’s one good use case for this and for text to speech. We’re just releasing it so we’ll have support for that soon too. I think it’s similar use case.

Voice Personalization

Pavan: It’s little different the kind of things that you want to extend a [00:25:00] text to speech model to, which could be like voice personalization, voice adaptation for enterprises.

Many enterprises need very specific kind of tone, very specific kind of like personality for this kind of voice. And all of those are like good use cases for fine tuning.

swyx: This one I was gonna ask you, we never talked about cloning voice clothing here. How important is it, right?

Like I can clone a famous person’s voice. Okay. But

Pavan: the main use case would be like for enterprise personalization, like enterprises need like a lot of customization. You don’t want the same. Voice for all the enterprises. Each enterprise want a customized, specialized something which is representative both their brand and also their, I guess safety considerations and the use case I think the kind of thing that you would deploy as a empathetic assistant in the context of a healthcare domain would be very different from the kind of thing that would be in a customer support bot and would be different from like more conversational aspects.

I think those are the. [00:26:00] Customizations you would expect from enterprise. And that’s the main use case, at least from our side.

Vibhu: My, my basic example is you don’t want to call to customer services and have the same exact voice. It’s just, it’s gonna be weird.

Long-Form Speech Models

Long-Form Speech Models

Vibhu: But also on the technical side of this, so there’s like a few things in TRO that I thought were pretty interesting.

He’s a big fan of this paper. Oh, he said very good paper. He said this is the best SR paper he’s ever read. Yeah. I’ve hyped up this voice paper enough. We covered it. Somewhere, but a big thing. So Whisper is known for 32nd generation a 32nd processing. You extended this to 40 minutes. There was a lot of good detail in the paper about how this was done.

Even little niches of how the padding is. So it’s very much needed. You need to have that padding in there, the synthetic data generation around this. I’m wondering if you can share the same about the new speech to text, right? Text to speech. So how do you. How do you generate long form, coherent?

How do you generate, how do you do that? And then any gems? Is there gonna be a paper?

Pavan: Yeah. Yeah. They would be a technical report. Okay. Yeah. I think I could have a lot of details.

Real-Time Encoder Advances

Pavan: But me I think the [00:27:00] summary of it, actually, some of the considerations in this paper were, because we started with the wipa encoder as the starting point, and now we have in-house encoders, like the bigger time model, for instance, which we released in January.

Also release a technical report for that real time model as well, which is this dual stream architecture. It’s an interesting architecture. You should check it out. And there we have a causal encoder and I don’t think there’s any strong, multilingual causal encoder out in the community. So we thought it’s a good contribution.

So that’s one nice encoder there. Other people want to adapt. That’s a good end code. And we train it from scratch. I think her. Post stack is now mature enough that we are able to train super strong ENC codes. And some of these considerations, like spatting and stuff, is a function of the Whisper ENC code.

And now that we train encoders, inhouse the design concentrations are different.

Scaling Context for TTS

Pavan: And for the question on text to speech, I think that’s also leans onto the original auto aggressive decoder backbone. I think, it says very, almost identical considerations. I think the long context in it’s not even long con, [00:28:00] so the model processes audio at 12.5 herds, so one second maps to like 12.5 tokens.

So I think one minute is like 7.8 tokens. You can get like up to 10 minutes in eight K context window and get half an hour and 30 K context window. So that’s and 30 2K context is something that’s we are very comfortable training on. We can extend it even much longer. 1 48 K. Okay. You can naturally see how it can extend to even our long generations.

Yeah. We need the. Like data recipe and the whole algorithm to work coherently enough through such long context. But the techniques are some way very similar to the text, long context modeling. And the key differences, it’s just doing flow matching order regressively instead of a text open prediction.

swyx: Okay. I think that was most, most of the sort of voice questions that we had. But

What Makes a Model Small

Vibhu: I have a big question on Mr. Al, Mr. Small. So what is small? How do we define [00:29:00] small? What is this? What is this? I remember the days of Misal seven B on my laptop. The snuff fitting on my laptop. I could run it on the big laptop, but

Guillaume: it’s just additional.

Question of terminology, like here what we did, baseball is north active parameters, but it’s true. Really not give it another name, but yeah, we could have called it medium, but only, I,

I suppose it’s a model that we released mixture of experts. It’s a model that combines different model before which we were doing the same, is that we had one model, general model for Israel. Doing instruction following, were like a separate model that was Devrel trial. So qu coding specify specific to code with another model for Reason Maal.

So this were separate artifacts built by different team at trial on what we’re doing is basically merging all of this. It was, you had pixel trial was the first vision model. We was like a separate model on the way we do things internally is that we have one team focus on one capability, build one model.

On the means mature, mature enough, we decide to merge this into the [00:30:00] matrix. But here it was the first time we basically match all of this into one. But there are some other things we did at first time to merge time, for instance, like more capabilities or function coding I think would be, are, it’s going to be much, much better in this trial, small platform.

But but yeah, so it’s our latest model on the working is,

Vibhu: and yeah, key things is it’s very sparse. Six, be active pretty efficient to serve. 2 56 K context. Yeah,

Merging Capabilities vs Specialists

swyx: I think what’s interesting is just this general theory of developing individual capabilities in different teams and then merging them.

Where is this going gonna end up?

Vibhu: Like we’ve seen the five things put together in this. Yeah. What are the next five teams?

swyx: I think actually OpenAI has gone away from the original four Oh. Vision of the Omni model. This was what they were selling. All modalities and all modalities out.

But I feel like you might do it.

Guillaume: I think there’s some mod where it’s not competitive use, for instance for audio. For audio here, if you want to do transcription, I think it makes no sense to use a model. If you just want to trans tech it, it’ll be very inefficient. If you want to do audio, you probably just want to be the [00:31:00] one VR 3D model performance essentially

swyx: the same.

It’s going to be incredibly cheaper. So here, that’s why we want

Guillaume: to have a separate but just does this. Yeah, I think the question is just, yeah. If you are to, to your model. By speech and you asking like a very complex questions on how you do this on the, just to cascade things. Do you want to put a d in a model that has like a one key around it?

It’s like a, not a competitive discussion, I think unaware if you doing into the direction, but that’s possible. Of course. But yeah. But I think for us, the next capabilities we want to try to integrate into these models when we are going to be yes, like marketing or no reasoning better, I think more capabilities that people don’t talk too much about, but at high bottom, I think for our customers in our, on different industries, for instance, things are around like a legal computer.

I design all these things that is this males out of the box are to put at that. Because people, if you don’t prioritize this, there is not like too benchmark on that. But

swyx: this done how to

Guillaume: make this good and this just start to do the work. Extracting some that processing it [00:32:00] expression. So yeah.

But we are offering the imagine to this.

swyx: I think for voice. Yeah. The key thing I think over maybe like the last year or so with VO and gr Imagine and all these things is joining voice with video, right? Which people don’t understand spatial audio because like most TTS is just oh, I’m speaking to a microphone in perfect studio quality.

But when you have video, like the voice moves around.

Pavan: That’s true. The constitution was a little different in the sense that there it’s like a a standalone artifact where you get the whole thing and you consume it. But in a conversational setting, it’s a, you need the extreme low latency.

swyx: Yeah,

Pavan: streaming would be one of the primary concentrations.

swyx: You can build a giant company just doing that, right? So you don’t need to do the voice, but I was just know on the theme of merging modalities, that is something I, I am like, wow. Like I didn’t, everyone up till, let’s say mid last year was just doing these like pipelines of okay, we’ll stitch a TTS model with a voice thing and a lip sync [00:33:00] thing and what have you.

Nope. Just giant model. Yeah.

Open Source Mission

Vibhu: I have a two part question. So one is, it’s still open. It seems like open source is still very core to what you guys do and I just have to plug your paper. Jan 2024. This is the one trial of experts like. Very fundamental research on how to do good.

Moes paper comes out very good paper for anyone. That’s just side tangent. No.

swyx: This thing caused, we bring back, eight by 22 was like the nuclear bomb for open source. I think it takes Shouldn be more seven B more. Yeah. Yeah. But this is a bigger opposite than me.

Yeah. Yeah I don’t remember this. I remember, I don’t think it was January, right? It was like new reps it was, it dropped during new reps and everyone in Europes was December of 25th, I think. Yeah. The model was did as well.

Vibhu: It’s just a little update probably.

swyx: Yeah. No, but you have a point to make.

Vibhu: No, you gotta check that. But then, I just want to hear more broadly on open source for you guys, and when you had asked earlier [00:34:00] about what’s next, what are the other, side tapes working on you. You put out Lean straw. This,

swyx: it’s not necessarily surprise. I was like, I don’t, this doesn’t fit my mental model or Misra.

Guillaume: Yeah. First for open source in general, I think it’s really something which looks to the January of the company. I think we started it per once, is we so we have open sourcing with, since the beginning and even before this. So before this, so me and Tim were at Meta, we released LA and I think what was really nice.

To see that before this, for most researchers like universities, it was impossible to work on elements. There was no alien outside. And if you look at many of the techniques that were developed after, for instance, was open source all this post-training approaches like even DPOD, like preference optimization, all of this were done by people that had access to this portal.

And it’ll have been impossible to do without this. So it’s really making sense, move faster. So we really want to contribute to this ecosystem. I think like the deep and also like very lot of impact. All these papers that are I think in the open source community are really helping the science community as a whole to move faster.

So [00:35:00] we want contribute to this ecosystem. That’s why we’re releasing very detailed technical reports. So ma trial and our first reason model, and ation, lot of results, things that work, things that did not work as well. Think helpful on the, yeah, so for the audio model also to share a lot of details, share of them for real time model.

And the, yeah, so we really want to continue this, basically belong to this community of people who share science. I think we really don’t want to be, leading in a world where the smartest model, the best models are only behind, close doors. Only accessible to a shoe companies that we, as a power to decide we can use them on it.

I think it’s a scary future. We don’t want to live in, we really want this model to be accessible to anyone that want. Intelligence to be used unaccessible by anyone who can use it. So yeah, so that’s why we are pushing this mission and source model. Yeah. So not, so yeah, no strategy. So it’s open source, not the first model, so not the best on the Yeah.

Lean and Formal Proofs

Guillaume: LIN trial I think is also one step into this direction. So it’s yeah, a bit different than what we are usually releasing. But we have a small team internally [00:36:00] working on them. Formal proofing, formal math. So I think a subject we care about in general and we were working on reasoning. I think we started too early before doing reasoning without LMD is very hard, especially when you work with formal systems because the amount of data you have is negligible.

It’s addressable community of people writing like formal proofs. But the reason why we like it is because I think there is if you look at what people are doing with reasoning, is there, the problems that you can use. Are usually going to be problems where you can verify the output. So for instance, all this ai ME problem where the solution is a number between 100, like a thousand.

So you can verify, compare this with a reference or it’s an expression. You can actually compare the output expression generic with the reference. But there are many, most of them have problem and most of the reason problem. There is no like way to easily verify the solution. If the question is show that F is continuous, cannot compare in the reference, right?

If it’s a probe that this is true or probes is properties, there is no way to. You cannot act, simply verify the correctness of your proof. So it’s hard to apply the, there is no referable reward here. So [00:37:00] what you could provide is of course, like a judge and judge that will look at your proof. But it’s very hard and it’s very, you could do certain, some reward hacking happening there.

So it’s difficult. You could provide like a reference proof, but then there are also many ways to prove the same thing. So if the model says give negative reward because it’s a different poop, maybe it was still digit proof, just different. So it’s not going to work well. What’s nice with lean and with formal probing is that you don’t have to worry about this whatsoever.

We just,

swyx: they’re all function is largely compiles in lean is functionally the same. Exactly.

Guillaume: It’s like a problem if it compiles it’s correct. It’s very easy. And you can apply this and then you can,

swyx: it’s just way too small. So no human will actually go and do it.

Guillaume: Yeah, that’s exactly.

It’s the only people can do it. It’s like a very small committee of people doing a PhD on that. So it’s super small. And it’s sad because it’s actually very useful on not just mat, but also in software verification. So for instance, software verification today. So tiny market. Very few industries work on this and we need that.

It’s usually going to be like companies like building airplanes, air robotics,

swyx: like

Guillaume: things [00:38:00] where they absolutely want to be sure. Life depend on this, but it’s very rare that people formally verify the correctness of their software. But I think one of the reasons for this is simply that it’s just hard to do.

swyx: Are you think of TLA plus? It’s the language that some people do for software verification? No. That people use in a ference, but but yeah, it’s the reason I think why people don’t use it more and why this industry is not as big as could be is because it’s very hard. But now with cutting edges that are there, it’s going to be very different.

Guillaume: We’re going to see much more of this. So I think yes, industry there is going to be much larger in the future that we, these models. So yeah. Here also anticipating this a little bit, we wanted to work on that because it’s proving like a math theory and like a, essentially the same tools.

swyx: Yeah.

Reasoning Transfer and Agents

swyx: One of my theories is that because the proofs takes so long, it’s actually just a proxy for long horizon reasoning and coherence and planning. Maybe a lot of people will say okay, it’s for people who like math. It’s for being okay. It’s like a niche math language. Who cares? But actually, and you use this as part of your data mixture for [00:39:00] post-training and reasoning, actually, it might spike everywhere else.

Yeah. And I think that’s un under explored or no one’s like really put out a definitive paper on how this generalizes.

Guillaume: Yeah, absolutely. And

Pavan: I think even

Guillaume: that’s what we’re seeing already. For instance, you should do some reasoning on math as then the American should do reason even.

Yeah. In the early stage. So we, the, there is some transfer, some sort of emergence that happens. And I think some, it’s also interesting, it’s not just I think the topic in general, but it’s, there is a lot of connection with this on including agents because. Sometimes the model can see like a three that it has to prove it’s very complex, but then it can take the initiative to say, I’m going to prove this three lr.

I’m going to suggest three Rs, and I’m going to in parallel prove each R. So three of them in parallel with sub agents, but I’m also going to prove them in theory and the three tool so you can do this also. Pretty interesting. You can, even if you fail to put one of the LeMar, you can actually, maybe you succeed to put the normal lema too, so you get some possible reward here.

So it’s a bit less Spartan issue, just get to zero one for the entire thing. [00:40:00] So it’s pretty interesting. I think we can actually,

Vibhu: yeah, it’s also an interesting case just for specialized models in general, right? Like the cost thing you show is pretty interesting yeah, similar score wise, you are, thirty, seventy, a hundred fifty, three hundred bucks.

Smaller.

swyx: I think cost is a bit unfair, right? ‘cause this one is at like inference cost. It’s always there on top with their margins on top of it. But, we don’t know anything else, so we gotta figure it out.

Vibhu: Okay.

Next Frontiers in Training

Vibhu: I did wanna actually push on that more. Not on cost, but you mentioned about, okay, it’s a great way to have verifiable long context reasoning.

What are other frontiers that, I’m sure you guys are working on internally, there’s a lot of push of people pushing back on pre-training. Scaling, RL pushing, compute towards having more than half of your training budget. All on rl. Where are you guys seeing the frontier of research in that?

Guillaume: You mean the

Vibhu: just in foundation model training in the next, one thing that you guys do actually is you do fundamental research from the ground up, right? So you probably have a really good look at where you can [00:41:00] forecast this out.

Guillaume: Yeah. I think for us we’re still working a lot on the pre-training side.

I think we are very far from situational, the pre-training. I think ML four preprinting will be like big step compared to everything we have done before. So we are pretty excited about this. And I think on the other side, I think now we have more and more to think about this algorithm that will actually support this very long trajectories.

I think when it was, for instance, GRPO for it doesn’t really work this any bit of policy. Which was okay initially because you are solving math problem that can be solved in like a few thousand tokens. So the model can alize them pretty quickly. So when you do your update, the model is never too far off.

It’s never too far off. But now when you are moving towards this kind of problems where certain takes hours, like six hours to get a reward, then your model is co pick places. So you have bi new infrastructure that supports this, but also new A, so now everything we’re doing internally, we’re trying to. Build some infra that we actually anticipate is what we have in six months, one now, which is this extremely no scenarios on the, I think when we started Missal, part of me and [00:42:00] we wanted to, is very nice under element where people are there, they can do research, they like with a lot of resources.

So it was nice. I think things changed a lot when I think when J Pity came out. I think after that I think was. This one is same again. But but yeah, but it was nice. And I think we also want to work part of this descrip before

swyx: coming to the end.

Hiring and Team Footprint

swyx: We’re just, obviously, I think you guys are doing incredible work.

You’ve, they are a very impressive vision for open source and for voice. What are you hiring for? What’s the what are you looking for that you are trying to join the company?

Guillaume: Yeah, so we are hiring a lot of people in our sense team. We’re hiring, in all our offices. So we have a, our H two is in France in Paris.

We have a small team in London. We like a team in Pato as well. Co we open some offices in in SAU, in Poland. So one in Zurich. We also like some presence in New York as well on Sooner one in San Francisco. So we all bit either way also like hiring remotely. So we’re going the team trying to hire like very strong people.

I think we want to stay, so the team is not. Instead of fairly small team. [00:43:00] But I think we want to keep it that way. ‘Cause we we find it quite efficient. So like a small team they agile so yeah.

swyx: Okay.

AI for Science Partnerships

swyx: Let’s focus on science and the forward deployed. We actually are strong believers in science.

We started the our new science pod that focuses specifically on the air for science. What areas do you think are the most promis.

Guillaume: What we’re pretty excited about right now, and something we have already started doing or that we’d probably be able to share more about this in a couple of months, is that we are exploring AI for science.

And there are a lot of areas where we think that you could get some extremely promising buzz. If you were to apply AI in these domains. There are a lot of long inputs. You just have to find these domains where actually AI has not been yet applied, and it’s usually hard to do because the people working in those domains don’t necessarily know the capability of these models.

They don’t know. How I would just have to pair them with Yeah, exactly. Your researcher slashing, which is actually hard to do. But this matching, we’re doing it naturally with our customers. So we have some company we are very closely with. So for instance, ISM Andreesen are one of our partners, so we’re doing some research with them on their other, like tons of extremely interesting problems.

Columns in physics, in [00:44:00] science matter science that they’re essentially the only ones to work on. ‘cause they’re doing something No, no one else is doing on the, yeah. So there are many domains where AI can actually revolutionize things. Just you have to think about it on you familiar with what can do or to apply it.

So yeah, it’s something where more modeling with our partners, with our customers sort AI for s, but.

swyx: Yeah. Okay.

Forward Deployed Skills

swyx: And then for deployed what it makes a good four deployed engineer, what do they need? Where do people fail?

Guillaume: I think it’s usually you need people that are very familiar with the tech and not necessarily with a lot of research expertise, but that are actually pretty good at using this model that can actually like that know how to do functioning, that know how to like, start some error pipeline.

And it’s it’s not easy. It’s something that mucus. Majority of companies will not be able to do this on their own. So here I think we need people that are, that like to solve problems that are accept solving some complex, very concrete problem. It’s applied science basically.

And yeah, so I think it’s not too different. I think from the case you need in research because it’s essentially you are trying to find solutions to problems that in [00:45:00] customers have not yet. So sometimes it’s easy. Sometimes you’re here to do the work. You have to like create synthetic data.

Find some edge case. So it can be, yeah. Depends on the problem. But but yeah, you have to, I think it also a bit of patience on the be creative. I think very similar skill is Asian,

Pavan: the diversity of the work they do. It always surprises me. It’s it’s, it goes all the way from the kind of stuff they encounter in industries.

It’s just very interesting. I think.

swyx: Any fun like success anecdotes.

Guillaume: Yeah, it can be actually training this small model on edge that just we do one specific thing can be like training some very large model without some specific languages as well. Making models really good at some tube use, like for instance, computer ID design, these kind of things.

Is that pairing with vision as well? Yeah,

Pavan: and the fact detection for chips or like in, in factories identifying things like it, the. Diversity could be anything where you can deploy these foundation models. So yeah the work to make it work in that specific setting, basically whatever it takes to make it like add value in that, by the way, workflow.

Vibhu: Yeah. [00:46:00] And it goes across the stack, right? Like even just pulling up the website like.

swyx: It’s so broad on compute. It is so broad.

Vibhu: We didn’t even touch on if you have a coding CLI tool. One thing you guys were actually like, I think the first tool was agents, ral agents. You had the agent builder, you can serve it via API and all that.

And I’m guessing forward deploy people.

Guillaume: Yeah.

Vibhu: Help build that out and stuff.

Customer Feedback Loop

Guillaume: It is also why we are, so we’re doing many things, but I think that’s also part of the value proposition that sometime know customers. They’re always very. Extremely careful about their data and they don’t want to, they don’t like, trusting so many partners, trusting one partner for code, giving the data to another third party for like audios and another one.

So they don’t like this here. What they really like with our approach that we can help them on anything so they don’t have to send the data to so many clouds. So yeah,

swyx: I think that there can be many orders of magnitude more. F Ds then research scientists and they don’t need your full experience, but they’re still super variable to customers

Guillaume: in practice.

These two teams [00:47:00] are still quite intertwine, very often. Yeah. So first of all, they’re using the same tools, the same data pipeline and everything on the, it’s it’s very helpful for the science team to get the feedback and the solution team ‘cause they can. Look at these customers are trying to do this.

This is not working. It can really be show in the next version. Yeah. But this is basically a real world eval. Yeah, it’s real world eval and it’s not something, for instance, if you’re just working in the lab, it’s just ships model. But you don’t do this work of for customers. You have no idea for whether your model is good at this H case.

For instance, you even in year found this, right? So yeah, there is a very gap, big gap between the public benchmarks that are very like academic. On

Pavan: the rare cases are just very diverse and in the specific concept of a customer, you can fine tune and make it like first evaluate, create a solid eval, benchmark, and then measure in the context of their, the kind of audio.

Like for instance, one use case is literally just, there’s the word for kids and they have to just say it out. It’s a very specific thing. You’re just saying one word and then you have to you, you’ll grade the kid whether they did it right or not. It’s [00:48:00] like R for, but so there’re very diverse use cases and the idea is that they, the.

Applied scientist engineer will go and make it better. And then from the learnings we incorporate it into the base model itself. So it’s it’s just better out of the box.

Vibhu: Yeah. It’s a good full circle system. Like the foundation model evals are all just proxies of what you really, you’re never gonna have one that says it, it doesn’t make sense for there to be, a one word transcription like that.

It’s not something you wanna fit on. Perfect.

Wrap Up and Thanks

swyx: Everyone should go check out everything that Michelle has to offer and try the TTS model, which will link in the show notes. But thank you so much for coming tha thanks. Such a stretch.

Materials science is the unsung hero of the science world. Behind every physical product you interact was decades of research into getting the properties of materials just right. Your gym clothes contain synthetic fibers developed over decades. The glass screen, diodes, and chip substrate technology needed to read this blog post were only viable due to many teams of material scientists.

Our guest Prof. Heather Kulik was one of the first material scientists to realize that there was alpha in combining computational tools with data driven modeling — she did AI for science before it was cool. She has a hard-fought perspective for how to succeed in this field. Yes, she believes the wins are real. To get there you must work hard to deeply integrate domain expertise with AI techniques, and also maintain a discriminating mind. Ultimately what matters is you succeed in the lab, and nature doesn’t care about how hyped a model is. These lessons personally resonated with the Latent.Space Science team and our own experience.

This episode is a must watch for all aspiring AI for science practitioners. A few highlights:

Designing new polymers with AI: Heather’s group recently used AI to design new polymers that are significantly stronger. These materials were created and tested in the lab, and the scientists who built them were surprised by the designs. The AI had figured out certain building blocks could break in a novel way. The AI discovered a purely quantum mechanical effect, and after convincing their lab collaborators to actually synthesize it, the material turned out to be four times tougher!

The twenty-two-atom ligand challenge: When asked about the role and need of human scientists, Heather points out that AI has a strong understanding of academic chemistry, but is still lacking intuition. Every time an LLM is updated, Heather asks it to design a ligand that contains exactly twenty-two heavy atoms. She has yet to find one that can succeed at this seemingly simple task that any expert could do in a second! Is this the chemistry counterpart to counting ‘r’s in strawberry?

Side note: Heather joked that this comment would date itself immediately, so we decided to see if this was still true three months after recording. We found some interesting results! We asked both Claude and ChatGPT to design a 22 atom ligand for both a metal-organic framework (MOF) and a Kinase protein.

* For the Kinase, both models got it right: Claude pulled out RDKit in a python script and iterated on several designs, whereas ChatGPT just one-shotted it.

* For MOFs, both models got it wrong, generating ligands with 21, 23, or 24 atoms, yet stubbornly not getting 22 atoms.

Is there something different about how LLMs reason in the materials and bio domains?

Materials vs biology: The two biggest domains of AI in science have been biology and materials. We asked Heather if there could be an AlphaFold moment for materials. Her answer reframes how we should think about the field:

* First, the datasets in material science are woefully lacking in comparison to the bio world. The closest to ground truth in most cases are noisy DFT datasets. These are just approximations to the real world! The datasets that are accurate are all boring, as Heather quipped “We have really good datasets for really boring chemistry.” Furthermore, good experimental structures are hard to come by and require interpretation. So generating generating high-quality, novel datasets at scale would really drive the field forward.

* More philosophically, AlphaFold is making predictions in a fairly limited space: there are just twenty amino acids. Sure, even here AlphaFold doesn’t get everything right, but it seems plausible that one could learn the entire design space. For materials, each element is a new set of interactions and chemistry, with little to no transferability. This is a massive open problem in material science that we hope some of the smartest AI scientists will want to work on!

The difficulties of trusting the literature: Heather’s team has spent the last few years using NLP and later LLMs to extract data from literature. Even a few thousand data points from these papers can be valuable for guiding her group’s work. One surprising result: sometimes the reported values for a property (say temperature) do not match up with the graphs in the papers! So there’s lots of potential in using LLMs to mine data from the literature, just do it with care.

The role of academia in an ever-changing world: One theme that has been running through many of our conversations has been the changing role of the academic — and the scientist — in science. When startups are raising $100s of millions and hyperscalers and Big Pharma are all ramping up AI-for-science efforts, the academic researcher needs both resources and judgement about problems to chase more than ever.

Resources include data that is organized for machine learning, access to high throughput experimentation labs, and compute resources. These are all things that academics can build together. More importantly, Heather emphasizes curiosity about problems that haven’t hit the radar of the heavily capitalized AI companies. After so many years on the forefront of AI for Science, Heather’s judgement that Chemical Engineering and Material Science still need curious people asking questions with no clear path to money is a welcome beacon in the AI fog.

Full Video podcast

Is on Youtube!

Mar 23 update for Latent Spacenauts: this episode was recorded before the Dreamer team announced they were joining Meta Superintelligence Labs, and it turned out to be the last interview they did before the news became public. Consider this a snapshot from just before the transition!

In 2024, David Singleton left Stripe and joined forces with Hugo Barra for a buzzy stealth startup named /dev/agents. This month they emerged out as Dreamer, a consumer-first platform to discover, build, and use AI agents and agentic apps, centered on a personal “Sidekick” that helps users customize experiences via natural language.

Sidekick is nothing less than an “agent that builds agents”, with all the complexity that that entails:

You’ve seen many many website builder, app builder, and even agent builder startups by now, but our favorite detail is the sheer amount of work that has gone into the “full stack” nature of the platform, including shipping their own SDK, logging, database, prompt management, serverless functions, and so on. Most platforms restrict the tech stack you can use just to get off the ground — Dreamer does it “right” by letting you push whatever arbitrary code you want to their VMs.

Paying the Builders

Of course former leaders of Stripe and Android would not stop at just building the tools, but also building the ecosystem. Dreamer is deeply aware of the 4 sided network effect it has going on and is ready to fund all of it.

It’s time to Dream!

Full Video Episode

on youtube.

Transcript

[00:00:00] Meet Dreamer Purple

[00:00:00] swyx: Okay, we’re here in the studio with David Singleton. Welcome.

[00:00:08] David Singleton: Hey, Wix. It’s great to be here.

[00:00:09] swyx: It’s great to have you. Uh, we have very sympa that your company color is the same as Lean Spaces color.

[00:00:15] David Singleton: That’s right. Dreamer Purple.

[00:00:17] swyx: It used to be Devrel agents, which I thought was very cool. It’s like you call back to Devrel Payments.

[00:00:22] David Singleton: Yeah.

[00:00:22] swyx: And you were obviously CTO Stripe. And talk to me about just the origin or thinking process behind Dreamer. Yeah. And maybe, maybe start with like, what, what is Dreamer?

[00:00:31] David Singleton: Yeah.

[00:00:31] What Is Dreamer

[00:00:31] David Singleton: So Dreamer is a new product, uh, which everyone can come and play with today. Um, it’s a place where everyone, literally, everyone can discover, build, and enjoy and use AI agents and agenda apps.

[00:00:45] And we really did design it for consumers, for folks who are not necessarily. Uh, have any kind of technical background. It’s really aimed at everyone. I think often of my sister, she’s very smart. She’s not in the slightest bit technical. She has lots of problems in her life that [00:01:00] she would like to be able to have great software and intelligent software to solve.

[00:01:04] But you know, even with the rise of tools like Cloud Code and so forth, she’s got no way to get started. And Dreamer is a place where she can come in, grab some intelligent apps that other people in the community have built, start using them right away, and solve real problems in her life.

[00:01:19] Sidekick And Waitlist

[00:01:19] David Singleton: And at the core, we have a personal agent called the Sidekick.

[00:01:24] Um, you can give your sidekick a name, you can give it its own personality, and it really helps you across your entire day, your life. It helps you use all of the agents on the platform, and it also helps you build anything you want. And we’ve been working in this for a little while. We recently launched in beta.

[00:01:41] So anyone can go to dreamer.com, join the wait list. Um, and we have many, many, many people in the community now who are building really fun, really powerful, really useful. Agents and the agentic apps for themselves.

[00:01:54] swyx: I think we’re gonna go right into a demo. Yeah. I just wanna make an observation that, uh, you, you, [00:02:00] you put discover first before build.

[00:02:02] Mm-hmm. But actually, at least for the engineers in the audience. ‘cause we are primarily engineers and you’re primarily targeting consumers, right?

[00:02:08] David Singleton: Yeah.

[00:02:08] swyx: For engineers. Like, there’s a huge full stack of stuff, which we’re gonna dive into. Let’s write. It’s so impressive. I’m like, holy s**t, this, this is what I’ve always wanted.

[00:02:16] Cool. Uh, so, so I think that’s really good and I’ve, in some ways, I think given your background given, uh, Hugo’s, is it Hugo? Hugo.

[00:02:24] David Singleton: Hugo. Hugo Bar. Yeah.

[00:02:25] swyx: Hugo, it’s not surprising that you can basically kind of build an app store Yeah. For agents.

[00:02:30] David Singleton: Yeah. So Hugo was my co-founder. Yeah. Um, Hugo and I met with our other co-founder Nicholas Checkoff in the very early days of Android at Google, where we were building Google’s first mobile apps.

[00:02:41] Uh, we then contributed to very core pieces of Android itself. And you’re right, we were really excited about building two things. One, solving a bunch of problems. That this breakthrough technology here I’m talking about mobile needed to have solved in order to make it work for real people at scale. And then secondly, building this ecosystem, um, [00:03:00] of third party developers using the Play Store, um, and able to deliver way more value on the platform than we could have delivered on our own.

[00:03:08] And we think about Dreamer in exactly the same way. So I was working at Stripe, as you mentioned, and we had the opportunity to put some of the very first AI agent systems in the world into production. And from the moment we did the first of those, I was just struck with a strong sense of conviction that this is breakthrough technology that’s gonna change how all of us work with computers and phones and so forth, all of the, the technology in our lives, but.

[00:03:34] There’s a lot of problems to be solved, for real people to be able to make this approachable. Um, and it really is kind of a direct analog for what we were solving back in the early days of mobile apps at Google and, and Android. So it’s, it’s been fun to bring that to life.

[00:03:47] swyx: Yeah. Uh, let’s look at it.

[00:03:48] David Singleton: Yeah, let’s take a look.

[00:03:49] Dashboard And Daily Briefing

[00:03:49] David Singleton: So, uh, dreamer.com, this is our homepage. This is where you can come and, uh, watch some videos about what is here and sign up for the wait list. Once

[00:03:57] swyx: you, I, I just wanna say for those listening, ‘cause we have a lot, you [00:04:00] know, switch to YouTube, look at the animations. So much care.

[00:04:03] David Singleton: We, we really care about, uh, this product being fun.

[00:04:07] Uh, and, and interesting to use. Obviously a lot of people are using it to do real important stuff. You can do real work, uh, here, uh, but also you can build fun things too. Once you get off of our wait list, you’ll come into the product. The first thing that happens is you’ll have a conversation with your side cake, which is this little friendly, uh, character here.

[00:04:27] And psychic will seek to get to know you and understand you. What do you care about? And will help you discover and build your first AI agents or agentic apps. After that, you’re, you’re gonna have a dashboard. This is my dashboard. Everyone’s is different. Um, you can see I have a few things here. I have a feed.

[00:04:42] So a lot of our agents do things in the background when you’re not looking and the feed is how they let you know what they’ve been up to. I have, uh, some widgets, uh, from apps that I have built. Uh, this one is called Calendar Hero. Uh, this is something that I installed from the gallery. Uh, so built by someone in our community.

[00:04:59] It’s a [00:05:00] really powerful calendar app because for each of my meetings, if it’s with someone I don’t already know, well it’ll actually go off and research it, um, and give me both a history of my interactions with those people and also a bunch of, you know, public useful information to, to get started. One of the things I love about this particular app is that every day it generates a podcast, um, a daily briefing.

[00:05:24] And one of the things that we’ve done with the platform is we’ve made it possible for all the things that agents do to show up in places that you care about. So if you look over here, this is the screen in my phone, and if I go ahead and open my Apple Podcasts, you can see right here. Your Daily briefing podcast is ready.

[00:05:39] This was produced by an agent running in my Dreamer account, and it was very easy by scanning a QR code to connect it to my Apple podcast. That’s what I listened to in the car now every morning. Yeah. On my way to work.

[00:05:50] swyx: It, it

[00:05:50] David Singleton: preps me for, for my day.

[00:05:52] swyx: So one additional bit of context. I asked you immediately after seeing this was like, what, what about, I wanna talk back to my agent and you said you actually started with voice and then you went to [00:06:00] podcasts.

[00:06:00] ‘cause it’s nice to have it pre downloaded

[00:06:02] David Singleton: that, right? That’s right. Um, yeah, we, you, you can talk to your sidekick. So, you know, on mobile we have, uh, a dreamer app and you can talk to the sidekick right here. Um, but we’ve actually found that making things, uh, show up in the other apps that you already use in your life is incredibly powerful.

[00:06:19] So let’s take a look at what’s kind of under the hood here.

[00:06:21] Gallery Tools And Payouts

[00:06:21] David Singleton: So I already mentioned that we have a gallery, so this is where you’ll find a lot of agents from our community. Uh, there’s. Many at this point, hundreds. And they are solving all kinds of, uh, use cases. I’d say the the top use cases are on personal productivity, but also a lot of information management that can range from personal information like docs and so forth, managing your emails.

[00:06:42] It also ranges out to public information that you might be interested in, but you need something to help manage the, the kind of fire hose of stuff that’s coming at you. For instance, I have, um, an agent which looks at all the AI news, um, all the time. There’s a lot of it and it finds the stuff that I would actually be [00:07:00] interested in, um, and I find it incredibly useful.

[00:07:03] So these are agents that you can install that other people have built. Anything that you install on Dreamer, you can actually just say, I wanna start making some changes, and we’ll look at that in a second. But in natural language, with the sidekicks help, you can change any of these experiences to work just the way you want them.

[00:07:18] But the base layer of the system are tools. So you know, as well as anyone swyx, that any AI system is only as good as the quality of data that it can pull in and the quality of action it can take. So before we launched our beta, we worked very hard to make sure that we seeded our tools with a bunch of very high quality and powerful integrations.

[00:07:39] So, you know, for instance, this is real Google search, this is actual Gmail. Um, and you can do very useful things with those. But also this is a platform for everyone. And as we got started talking to people in our alpha community, a whole bunch of sports use cases popped out and we realized if you want to build something cool for sports with ai, you need really high quality live data.

[00:07:58] So look at these [00:08:00] Formula one M-L-B-N-F-L, uh, these are tools, uh, that we’ve built. We’ve done a, these are not data scraped off the web. This is a, a direct data feed integration. And because it’s live and ‘cause it’s high quality, you can build really powerful stuff. But tools is not something that we are just going to kind of control ourselves.

[00:08:19] The platform is open for tool Builders to contribute tools that anyone on Dreamer can use. So, um, this is actually the place in the platform where I think software engineers, um, well number one, would love for you to come and play with it. Uh, but software engineers are really gonna build, um, a lot of powerful stuff into the system.

[00:08:38] And we are actually sharing something for the first time on this podcast, which there is, uh, tool builders on Dreamer get paid. So if you publish a tool to the platform and a lot of agents use it, you’ll actually get paid, uh, in proportion to their usage. And we’d love for folks to come and give this a try.

[00:08:54] We’ve got good docs that help you get started and you can build things that, you know, scratch your own itch. For instance, someone built this [00:09:00] Ski Bum tool, which provides live snow conditions for a bunch of, uh, ski resorts. I’d love to show you how I’ve used that in a second. And also we have some tools, partners where the tools themselves are paper use.

[00:09:12] So for instance, parallel web systems is a premium tool. Uh, you can do really cool stuff with it. Um, it’s a a, an agentic web research tool. And that one, because it’s expensive to operate, is paid on a, on a per usage basis. But if you’re coming in to build agents on the platform, even the premium tools, you get a free trial.

[00:09:29] So you get a chance to actually try them out, make sure that the use case is good for you before you decide to, to to sign up. So that’s tools. So we have the gallery, we have tools, and then the sidekick helps us put all of this together to build agents. We do that in the agents studio. You can also do this on your phone, but if I open up Agent Studio here on Desktop psychic’s, just gonna start a conversation about what you want to build together.

[00:09:51] I’d love to show you one that I made recently.

[00:09:53] swyx: Let’s do

[00:09:53] David Singleton: it.

[00:09:53] Building A Conference App

[00:09:53] David Singleton: Um, let’s look at something that hopefully is kind of near and dear to your heart. So one of the things I love about Dreamer and this kind of moment in technology is that if you think about it. There are all these things in your life where, have you ever gone to a conference?

[00:10:09] I know you have. Right? And, uh, big conferences have apps. Um, and these apps are usually built by agencies and they’re, they’re usually actually quite expensive to build. I’ve been involved in running some of these myself. And how many conferences have you been to where the app was good? Zero. Honestly.

[00:10:23] swyx: Exactly. Zero,

[00:10:24] David Singleton: maybe one. I, I’ve, I’ve been to one conference. That was pretty good. Wait, wait session sessions. Um, but, but the point is, they’re rarely great pieces of software. Right. And they’re also expensive to build, but they’re, they’re interesting ‘cause they’re episodic, they last for this one thing. Um, and then they’re, they’re not relevant anymore.

[00:10:43] Um,

[00:10:43] swyx: and so it’s the worst feeling to invest in them because, you know, it’s like, it’s got a limited. Date?

[00:10:48] David Singleton: Absolutely. So I decided to build, uh, a conference app for your AI engineer conference. Amazing. Uh, on Dreamer. One of the things that Swix has done, uh, which I [00:11:00] thought was very forward-looking, is actually put a whole bunch of data about the conference on the webpage in an LLM readable way.

[00:11:06] There’s an LLMs txt file, there’s a feed of all of the sessions in js, ON. So I used the data from your conference last year and built this intelligent app, uh, just by talking to our sidekick, uh, in Dreamer. So just to give you a quick tour, this is my Dream Conference app. What I always wanna do for conferences is I wanna be able to search for speakers.

[00:11:28] I’m usually there because, uh, there, uh, is a speaker I care about. So, you know, SWIX, you’re the speaker I care about. I can actually see here who you’re on stage with. So here’s, here’s Greg Brockman. You’ve read even ai, uh, and this is his session. And look Greg and Swix for the speaker. So let’s add that to my schedule.

[00:11:45] Great. And then maybe there’s a couple others I might see here. Like on day two, I remember there were some keynotes. So, uh, building the open agenda web, that sounds fun. So I add that to my schedule.

[00:11:55] swyx: She’s now CEO of Xbox.

[00:11:56] David Singleton: Awesome.

[00:11:57] swyx: Which is interesting. So cool. So,

[00:11:59] David Singleton: so I’ve [00:12:00] gone through and picked out a couple of sessions that I cared about.

[00:12:03] That’s as far as I usually get with any conference app. But of course you’ve got the whole of the rest of the conference to figure out what to do. So here is where the native intelligence of, of these things you build on Dreamer can come in. So I’m gonna click guide me. So Dreamers sidekick actually parsed out the whole schedule and figured out what some of the themes are and I can choose what I’m interested in here.

[00:12:23] I’m definitely interested in agents. Uh, I’m definitely interested in code generation and also reasoning in rl. So now I’m gonna say build my schedule. So what this is doing is. It’s going across every time slot for the conference. And it’s choosing among the things I could go to, which one it thinks is best for me based on my interests.

[00:12:41] It also uses its own memory of me that’s part of Dreamer, uh, to understand what I might like best. And you know, there’s an LLM prompt running for each one of these time slots. So this is, it’s not super fast, but it’ll be done in about 30 or 40 seconds. And I’m gonna have a special custom schedule for the conference.

[00:12:57] This, like I said, is my [00:13:00] dream conference app is exactly what I’ve always wanted and I was able to build this yesterday morning. Um, I did it between some meetings. I think I spent a total of 25 minutes of wall clock time on it. I did it over the course of a couple of hours. And, uh, here is my schedule for the conference.

[00:13:15] I can see it in a calendar view. This is what I should do on Tuesday, this is what I should do on Wednesday. Oof, no conflicts, but, you know, I may not go to every single thing. And there you have it built in, you know, dreamer. So let’s take a look at what the building experience actually looks like. So this is the, the actual account that I made it on.

[00:13:32] Oh, of course I should say anything you build on Dreamer also works on your phone. So, uh, here is my AI engineer conference app right here on my phone. Got all the same functionality, and of course this is the best place to jump into my schedule.

[00:13:46] swyx: Yeah.

[00:13:46] David Singleton: Um,

[00:13:46] swyx: so you could generate a podcast about it just completely multimodal, absolute thing, right?

[00:13:51] To me, I mean, this is why I outsource, I mean, well, I, I posted the L-M-T-X-T, the JSON because you cannot run an engineer conference in 2025 [00:14:00] and not let engineers. Do whatever they want.

[00:14:02] David Singleton: Yeah.

[00:14:03] swyx: And since all conference apps suck, I’m just gonna put up a ba minimum viable app and just let people do whatever they want.

[00:14:09] David Singleton: Totally. And the cool thing about this on Bremer is I published this to the gallery and you can use it so you’ve got one that’s built to my taste of conference apps. I think it’s pretty cool. But you might want something different. Yeah. In which case you just start telling the sidekick how to change it.

[00:14:23] So let’s just very quickly look

[00:14:24] swyx: at our, what sports grid is also, you can fork it, right? That I can publish. That’s right. I can publish your one and go, this is the base starter. It’s, it’s got good defaults, but go customize, whatever.

[00:14:32] David Singleton: That’s right. That’s right.

[00:14:33] swyx: Yeah.

[00:14:33] Agent Studio Under The Hood

[00:14:33] David Singleton: So let’s take a look at how I actually built this.

[00:14:34] This is real. So I’m gonna say make changes. This experience we’re looking at now is our, uh, agent development studio. Um, like I said, you can do this on your phone as well. And in fact, this one I started out on desktop. Let’s look at my actual prompts. I said, let’s make an agent called AI Engineer Schedule Planner should be a custom schedule planner for the AI engineer conference.

[00:14:53] I’m not gonna read this all up. You get, you get the point and it told it where to get the data from. So that was the first prompt. And actually after I gave it that [00:15:00] prompt, I actually had a simple version of this app working, um, after the sidekick took one turn. So the Sidekick is a, like a professional software engineer, and we’ve worked very hard to make this work and build functional apps for folks that might not have any engineering experience whatsoever.

[00:15:14] So, you know, done here we have build logs that are technical, but you can hide those away. And sidekick, as it is building, will actually translate everything that is coming out of, uh, of the, the harness into English that you can actually read. And by the way, this English is in the personality of your sidekick, which is fun.

[00:15:32] Um. And the way that we build agents and agent apps, it’s a little different to what you might have seen in some other platforms for a couple of reasons. One, just the build process. The very first thing that Sidekick does, it understands all the agents you’ve got set up. It understands all the tools and it will come up with a plan for how to realize your goal, how to make sure it actually has the data and the capabilities to complete it.

[00:15:54] It will occasionally refuse. If it can’t do what you’re asking, it will tell you I can’t do that. It needs another tool. And that’s a good [00:16:00] jumping off point for any of the tool builders out there to build a new tool. So it’ll fi first figure out how, then it will build it, and then it will actually test it.

[00:16:07] So it will actually make sure that the thing that it has generated is realizing your goal. And you probably know as well as anybody that anytime you can get any. Modern state-of-the-art coding model into a loop where it can make changes and perceive its own output and then fix bugs. Magic happens. So these builds, the first build will often take 10 to 15 minutes on Dreamer, which is a little bit longer than you might’ve seen on some other platforms.

[00:16:31] But the first thing that it creates will work most of the time. And then of course, as you start making smaller changes, you can like ask it to tweak the UI in any way that you like. Those are much faster. And just to give you a sense, uh, for this one, here’s something I asked. Put a logo, I gave it a logo file in static files.

[00:16:48] Use that as the title. So for folks that actually really want to dig, uh, into a bit more detail, we’ve provided a powerful IDE here. So I can actually see here’s the code that was generated and some pieces of the [00:17:00] code are more accessible than others, like the prompts. So this is the prompt that’s used by a powerful LLM in order to do that schedule picking.

[00:17:08] And I can actually read it here directly. I can edit it without having to ask the sidekick if I want to do that.

[00:17:12] swyx: So this is very nice.

[00:17:13] David Singleton: This is for the more, the more, uh, sophisticated users.

[00:17:16] swyx: Yeah. This is other people’s entire startup is prop management.

[00:17:21] David Singleton: This is true. The other thing that is different about Dreamer is once you’ve built something here, it’s ready to go.

[00:17:28] We host it. So you don’t have to worry about getting a database from a database provider signing up, getting API keys. You don’t have to worry about your LLM provider tokens. All of that is hosted on the platform. And you can use it yourself. You can share it to the gallery for other people to, to riff on it.

[00:17:46] You can also share it with your friends and coworkers to use your instance of the agent or agentic app. And we’re seeing that happen a lot in our community. We’ve seen a whole bunch of folks who built little applications for their personal life [00:18:00] and shared them with their significant other. We’ve seen people who are building little productivity apps for their team at work and sharing it, uh, among them.

[00:18:07] And we actually do this a lot inside of the company. So at this point we, we pretty much run the company on Dreamer agents for all kinds of important things. Uh, maybe a good example of that is, um, our wait list. People are signing up every time someone signs up for our wait list. A dreamer agent will actually research, uh, that person.

[00:18:25] And we’re looking for folks who are builders, not super technical to build agents and come in, uh, and give us a lot of feedback and we’re prioritized bringing those people off of the wait list First,

[00:18:35] swyx: just a quick question on that one is there’s, it may not come up again. Do you find enrichment APIs to be useful like the ZoomInfo?

[00:18:42] Uh, clear bit

[00:18:43] David Singleton: enrichment is a very, uh, common use case. Um, on dreamer. Any application on Dreamer can kick off a sub-agent to do a particular task. Um, so this actually is a powerful agentic harness that runs inside of its own [00:19:00] vm. Uh, we call them sidekick tasks ‘cause they actually run in the context of the sidekick.

[00:19:04] I’ll talk more about Sidekick in a second and. Enrichment is a very common use case. And the cool thing about a sidekick task is that it has access to all the tools on the platform, but also public data as well. And so very frequently enrichment on our platform happens using public data that it can be found in the web.

[00:19:24] There are some tools for getting people data, uh, from, uh, from various bespoke systems. And so that works pretty well. But actually, you’d be surprised. I mean, we would love if someone out there would like to build a ZoomInfo tool, we don’t have one today. We’d love to see that on the platform, and I’m sure it’ll be very powerful.

[00:19:39] But we’re also seeing that this powerful agent harness can pull a lot of data in on that note of tools that make experiences better, we’re constantly adding more tools because people in the community are building them and publishing them. We review the tools carefully and then they go live for everybody.

[00:19:54] Yesterday we added granola. And that was pretty cool. So I was talking to actually, uh, Sarah on my team was [00:20:00] talking to, uh, someone building on the platform this morning and they actually, they have an agentic app that they built, which is a kind of magic to-do list. So they put stuff on their to-do list and for each thing it kicks off one of these, uh, sidekick tasks to figure out how to move the ball forward thing.

[00:20:14] Sometimes it’ll complete it

[00:20:15] swyx: entirely. Yeah.

[00:20:16] David Singleton: Often by calling another agent on the platform and sometimes it just kind of researches it and helps ‘em take the first step.

[00:20:21] swyx: Yeah. Do you know, this is Sam Altman’s number one, ask for an AI app. It’s the self-completing to-do list.

[00:20:26] David Singleton: Yeah. The self-completing to-do list is something that a lot of people have built on Dreamer and are getting a lot of use out of.

[00:20:32] Yeah. And, and finding it actually genuinely I shouldn’t, I should, I should try that. Mm-hmm. Please do. And you’ll even find some in the gallery that you can remix. So he was saying this morning that he’s, he built this self completing to-do list, uh, on Dreamer already. But he connected the granola tool yesterday and now something really magical happens, which is when he says in meetings that he’s gonna do a thing, it magically shows up on his to-do list and then it can magically get completed.

[00:20:56] And then, as I mentioned, all the agents, all the [00:21:00] apps on Dreamer can actually work together. So our coding agent, as it builds them, does something very special where it exposes the internals of each of the experiences to the system. And then Sidekick can manipulate those to get stuff done. So he has built another agent, which he uses for recruiting.

[00:21:18] It kind of keeps track of candidates and also it’s got a kinda mini CRM function, so he’s able to introduce candidates to each other. He told us this morning that something he’d committed to do in a meeting that was recorded on granola yesterday showed up in his magic to-do list and his magic to-do list.

[00:21:34] It was like introduce a person for recruiting, used his recruiting agent to get it done.

[00:21:39] swyx: Ah,

[00:21:39] David Singleton: um, and this is, this is the dream. This is why we started the company. It really is the case that you can build and use these very powerful, bespoke experiences that can automate your life by working together. And I’d love to talk a little bit about how they work together.

[00:21:55] Ecosystem Trust And Monetization

[00:21:55] David Singleton: So obviously it’s really cool to have [00:22:00] software that will work on your behalf, but it’s only useful if you can trust it, right? So privacy and security is very important to us making these things accessible and. While also being trustworthy is hard. So the model that we have, which is working very well, is that the sidekick is at the core of everything here.

[00:22:22] So it is both your companion, your helper, but it’s also the traffic cup in the system. So when, when one agent wants to work with another agent and dreamer, it doesn’t do it directly, it does it via the sidekick, well ask the sidekick to do the thing. And the sidekick understands both everything, all the expectations that have been set with me as a user about what agents can do, which tools I’ve given them permission to use.

[00:22:45] And it will make sure that whatever is is going on is actually aligned with my own interests. And you know, that’s part of the background that I bring to this problem domain. I’ve. Worked for years, uh, keeping very important information, safe and secure. And [00:23:00] so as we started to think about this problem, we realized that we actually had to build something that’s a bit like an operating system.

[00:23:06] You know, the sidekicks, like the kernel, the agents and apps are like users. Yeah. Different rings. Exactly. Because if you try to pick off just one piece of this, you can’t actually make it work for people at scale. Uh, because you could build little vibe coded apps, but they’re gonna grab all your data willy-nilly.

[00:23:23] They won’t be able to work together. You actually have to invest in the fundamental core in order to make it work well for people. And that’s what we’ve been doing and it’s, uh, it’s been a lot of fun. One other thing I wanted to mention is, um, I’ve obviously talked about two things, tools and agentic apps.

[00:23:42] We really designed Dreamer to be an ecosystem and a platform, and one of my favorite quotes about platforms, I think it’s from Bill Gates, is that you can only be a platform. If you create more value for the folks participating and using the platform than, than the platform itself creates. [00:24:00] And that’s our goal here.

[00:24:01] So we at every step have been thinking about how do we make sure that other people are deriving even more value from Dreamer than we are? So in that vein, I already mentioned tool builders get paid and people can build agents that solve their needs and share them with others, and we are already thinking about ways that they can actually monetize those as well.

[00:24:24] Against that backdrop, one of the things that we are launching today is our Builders in Residence program. So there are tons of people building really cool stuff and contributing it to the gallery already, but we’ve been really inspired by programs we’ve seen at other companies where artists might be in residence, people that are very creative.

[00:24:43] And might have ideas outside of what the, the folks at the company or in the ecosystem already have. And so we are looking for creative people who have fun ideas and, you know, want to really figure out how to apply their creativity at the cutting edge [00:25:00] of technology today to come and work with us. So, uh, if you go to dreamer.com/latent space, you’ll find, ooh, well, we love Latent space.

[00:25:09] Uh, you’ll find a link both to, uh, our tool Builder information and our builder in residence program. And for builders and residents, we’ll let you in off the wait list quickly, build an agent, and then for a small number of, of the most creative folks, we’re going to pay you to build agents. Uh, you can work directly with our team.

[00:25:29] You know, this is like building Legos. So, you know, we’ve got some of the basic blocks together already, but if you need a Ron steering wheel and we don’t have one already, like we’ll build it for you. Yeah. Um, we really want to be inspired by, by these, uh, these builders in residence.

[00:25:43] swyx: This Legos thing is pretty common as an analogy.

[00:25:46] And there’s a, there’s a thing I call the master builder. Uh, we, the actual Lego company has master builders that they employ Yeah. To inspire people and post on socials.

[00:25:56] David Singleton: That is exactly what inspired us as well. Honestly, we talked about the Lego Master [00:26:00] Builder program, so that’s our builder in residence program.

[00:26:02] swyx: Yeah.

[00:26:03] David Singleton: Um, and then, uh, finally back on, on tools. Like I said, anyone can come in and build tools today. If you follow the latent space link dreamer.com/latent space, again, we’ll get you off. Directly off the wait list. So you can build right away, you can monetize by publishing onto the platform. That’s for everyone, the very best tool that gets added to the platform by mid-April.

[00:26:23] Uh, we have a $10,000 prize that we want to give out really, because we just want to seed the creativity of everyone out there. So we’re excited to do that.

[00:26:31] swyx: Yeah. And you know, uh, this is completely a flywheel, right? Like the more tools, the more builders, the more the third thing agents, you know, it just feeds into each other.

[00:26:39] David Singleton: That’s right.

[00:26:39] swyx: Yeah. Just on the payments thing, because we probably won’t touch on that again, but I have to ask the former CTO Stripe on payments as presumably you’re using Stripe Connect.

[00:26:48] David Singleton: Yeah.

[00:26:48] swyx: Um. Any pain points that you’re, people are very interested in agent commerce and micropayment and all these things.

[00:26:55] Presumably stable coins get into a conversation at some point, but maybe not now.

[00:26:58] David Singleton: Yeah, we are [00:27:00] really, really excited about e agent commerce. The first step we are taking is help people in the world who have never been able to build these kind of experiences and software before to build stuff that meets their passions, share it with the world and get paid.

[00:27:14] So that’s all commerce that happens on our platform, and so we don’t need anything new to facilitate that. Stripe Connect has existed for quite a while and is the perfect solution for this kind of stuff, so, um, we we’re excited about that. First and foremost, however. A lot of the things that people are already doing on Dreamer, we just talked about a self-completing to-do list.

[00:27:34] A lot of the ways that you want to complete to-dos is by actually closing the loop in the real world, and that’s going to involve the exchange of value. So we have some folks that are building tools already that actually do have money move in order to, to complete that, that loop. So far, we just want to be open and agnostic to all the protocols out there.

[00:27:54] I honestly think this moment in time is a little bit like the early web. So I personally started coding as a kid [00:28:00] and I think I got access to the internet in about 19 95, 19 96. And back then, uh, the web existed, you know, HTTP was a protocol, but there were also other protocols I was using all the time, like Gopher and UUCP and uh, various others.

[00:28:15] So the point is like the web, HTTP and HTML. Was just one among many protocols. And of course it became the winner and it’s awesome. Yeah. Um, but the others were also kind of interesting and viable at the time as well. And I think the world of agentic commerce is like this right now. Also,

[00:28:30] swyx: acp.

[00:28:31] David Singleton: Acp, exactly.

[00:28:32] All the, all the cps, you know, on Dreamer. We hope that folks will build tools that kinda make use of all of these things, but I’m sure that at a certain point. One or two will emerge as the winners, and then we’ll be able to build like really deep support in,

[00:28:44] swyx: yeah. This is like maybe a complete tangent, but I do think about how a lot of these companies in AI companies in particular have to switch from c based to usage based because of course, but then, then they end up, end up having to sort of [00:29:00] obscure the margins a little bit and then they inventing end up inventing their equivalent of rob robots.

[00:29:04] David Singleton: Mm-hmm.

[00:29:04] swyx: Uh, where they’re like, well, okay, well every company should have their own currency. And it’s, it’s like very short lead to a token.

[00:29:11] David Singleton: Yeah.

[00:29:11] swyx: Or, and I’m like, okay, well where does this end? I can’t really play out the next step as to like, is this chaos? Is this,

[00:29:18] David Singleton: yeah.

[00:29:18] swyx: Okay.

[00:29:18] David Singleton: Well, I think it is kind of like the wild west.

[00:29:21] I don’t mean that in a completely, it’s all completely disorganized way, but there’s just so many things that could happen from here. The Overton window is very wide, right? Not far how this might land. And I’m just very excited to be building a platform that can take advantage of all of those opportunities and we’re just gonna be there.

[00:29:36] Uh, working for our users to make sure that things that emerge work,

[00:29:39] swyx: you’re gonna own the consumers, you’re gonna be up the OS for the app store for everything.

[00:29:43] David Singleton: So one of the ways to think about this is, um, dreamer actually uses all of the state-of-the-art models as a user. You don’t have to think about should I be using, you know, Opus four six, or should I be using the five four model from [00:30:00] OpenAI?

[00:30:00] We are continually doing evals and so forth to make sure that the best things are there for you. You can just build on the platform and know that as the world ships around, you’re gonna get the right stuff for you. Um, and I think that’s something that is needed to actually have folks take advantage of this technology at scale.

[00:30:19] I’d love to show you another example of something I built.

[00:30:21] swyx: Let’s do it.

[00:30:22] David Singleton: This is another example of software that just lasts for a certain moment in time. So recently I went on a ski trip with a bunch of friends,

[00:30:31] ski

[00:30:31] David Singleton: Bum. Uh, so it uses ski bum. Yes. I went on a ski trip to Big Sky. I’d never been there before.

[00:30:38] And I made this little intelligent app for us. And you can see it says it’s loading big sky conditions. So it’s actually calling the Ski Bum tool that I just showed you, which is, uh, published in our, uh, in our gallery. So what is this? This is a little app that was just for our weekend trip. It shows the current status of all the lifts of Big Sky.

[00:30:54] Using that tool from the ecosystem, it shows the forecast for the upcoming weekend. It shows our [00:31:00] accommodation. This is just like where my group was staying. This is just for us and also a bunch of dining information that one of our friends, uh, put together who, who’s an expert on Big Sky. So I was able to take this app, share the link with my friends.

[00:31:12] They weren’t on Dreamer yet, just send it to them on iMessage and they get a version they can use on their phone. And of course, here’s the real kicker. So I’ve been on ski trips before and other weekend adventures with my friends. Yeah, people pay for different things and at the end of the weekend it’s always a pain to figure out who needs to pay, who to settle up.

[00:31:29] So we use this during the weekend. We added all of our expenses in here. Uh, too close are it’s drill data. It’s only too closely. And then at the end of the trip, we press split. And we’re, we settled up and we’re done. So there’s another dreamer. This was all through dreamer. So the, the actual payment? No, no.

[00:31:47] We, it happened because, because we paid for stuff in the real world, it was like, okay, this person needs to pay that person 20 bucks. Right? Right. This person already paid in that. Right. So it just helped us all settle up. We didn’t move the money on Dreamer. You could do that. And in fact, if you’re a tool builder [00:32:00] thinking about this and getting excited, like come build a tool to do that stuff.

[00:32:02] We really think of our tool builders as design partners.

[00:32:05] swyx: Yeah. I got, I got the tool. Uh, what, like, I hate, I use Bank of America. I hate bank, I hate the app. Mm-hmm. I hate the web. All banking websites just horrible.

[00:32:13] David Singleton: Yeah.

[00:32:13] swyx: So just build me, like build a thing on top of Plaid.

[00:32:15] David Singleton: Yeah. Right. And then just So

[00:32:17] swyx: five code by banking app,

[00:32:18] David Singleton: there’s already a tool for that.

[00:32:20] Oh. So, um, attain Finance is a tool, a builder in our community built. Okay. Um, and it uses a secure system like Plaid. To access your, uh, financial data and you can build powerful personal finance agents on Dreamer today using this tool. And like I said, we review tools carefully. So when bringing Attain Finance onto the platform, we did actually quite a detailed security review with that company to make sure that if folks build stuff with it, it’s, it’s gonna work well.

[00:32:49] So yeah, check that out. I think, uh, I’m, I’m pretty certain it connects to Bank of America. So you’ll be able to build the, the app that you wanted already?

[00:32:55] swyx: Yeah. There’s a couple of points I wanted to sort of dive in on, maybe highlight to folks, [00:33:00] because I, obviously, I spent more time with Dreamers. So we’re making a point where you choose on behalf of your users because they’re meant to be consumers.

[00:33:07] So maybe less technical,

[00:33:08] David Singleton: right?

[00:33:08] swyx: But obviously people can, how users can override. If you read that’s, but it’s not just lms, it is also the, the transcription. It, it’s like all, like there’s, there’s a first party curated set of here’s the house opinion. That’s right. On what?

[00:33:21] David Singleton: That’s

[00:33:21] swyx: right. The thing is, that’s right.

[00:33:22] Is what’s the list? Is there like,

[00:33:24] David Singleton: yeah, so actually if you look in the tool gallery, the first party kind of curated set are all the ones that have these grayscale icons. So we have a built in tool for image understanding, for image generation, for RSS, exploration, text to speech and so forth.

[00:33:38] swyx: Recipes.

[00:33:39] David Singleton: Uh, we actually do have a built in recipes tool.

[00:33:41] It turns out that a lot of people in our alpha wanted to do stuff for cooking. Yeah. Um, and you know, you can scrape the web to get good recipes, but we were able to quite quickly find a good repository of recipes. It works great here. Yeah.

[00:33:55] Stable Tool Interfaces

[00:33:55] David Singleton: So the point behind these though is that we’ll keep the interfaces stable, so they’ll always work.

[00:34:00] But you know, the best translation model and, you know, there are people using this translation tool to translate Chinese podcasts into English. It’s, it’s pretty powerful. It can deal with very long text, but the best translation tool today might be different from the best translation tool sometime next year.

[00:34:15] And we’re just gonna make sure that that translation tool is always pretty close to state of the art. So you can build something and you know it’s gonna continue to work well. Of course, some of our tools are branded. You may actually have a preferred way of buying groceries, like maybe you prefer Instacart and that’s great.

[00:34:29] You can use the Instacart tool specifically.

[00:34:31] swyx: Yeah.

[00:34:32] Partnerships And Ecosystem

[00:34:32] swyx: Your partnerships, uh, I mean, I don’t know if you ever hit of partnerships, but this is gonna be a bonanza for anyone on to do deals.

[00:34:38] David Singleton: We have an amazing person who, uh, works on all of our partnerships. Um, and it’s part of what you have to do to build a platform like this that’s gonna work for people.

[00:34:46] Like, we’ve gone and done that. Schlep has a lot of work, one talks lots of different companies, um, in order to make sure that you’ve got good tools at the core.

[00:34:54] swyx: Yeah.

[00:34:54] David Singleton: And then of course, because we’re open to tool builders contributing to the platform, this is only gonna get better and better and [00:35:00] better.

[00:35:00] swyx: Yeah.

[00:35:01] Agent Lab Routing Layer

[00:35:01] swyx: One observation I have this, this is gonna master a thesis I’ve been pursuing, which is, uh, what I’ve been calling an agent lab

[00:35:05] David Singleton: mm-hmm.

[00:35:06] swyx: Where you sort of different than a model lab in, in, in the sense that you never train your own models, but you are the router evaluation layer, ex subject domain expert for choosing between, uh, models.

[00:35:18] David Singleton: Yeah.

[00:35:18] swyx: And you’re explicitly doing these things. And so like in my sort of construction, every agent lab does some version of this where like, here’s the image understanding endpoint and we will route for you and don’t worry about it. Yeah. Sally, I think it’s kind of cool.

[00:35:32] David Singleton: I, I think it makes total sense. Um, and again, to make this work for folks that don’t follow the AI news every day, it’s an actually, it’s a, it’s a really important thing to do.

[00:35:42] Yeah. And it, it’s been, it’s been a real pleasure. I mean, I’m a, I’m personally a total geek for this stuff. I love it. And being able to go and dive into all those details in order to make it work well for other people. It’s a true pleasure. I cannot imagine working at anything else right now. It’s just so much fun.

[00:35:56] swyx: The tricky part is multimodality when some of these things do [00:36:00] merge.

[00:36:00] David Singleton: Mm-hmm.

[00:36:01] swyx: And you are, you’re sort of, this is your imposing structure on things that fundamentally don’t want to be structured. And so sometimes that might work against you, but for 99% of these cases, this is fine.

[00:36:10] David Singleton: Yeah. I mean, I think it’s gonna be very interesting to see how the, the, the world matures because a lot of the power of dreamer is the ability to kick off these subagents, so these powerful agent harnesses, which can actually change how they work based on the data.

[00:36:25] I actually think that we will be able to. Kind of keep up with and stay at the forefront of the changing landscape of how tools and systems work together. And that’s, that’s new. You know, software didn’t used to work like this and now it does. Um, so even, even just figuring out how to design the right pri to make that possible has itself be a lot of fun.

[00:36:44] Builders Can Publish Tools

[00:36:44] swyx: This is, is a sort of maybe two part question that why can’t streamer make its own tools? And then why don’t you let you builders maybe stand up their own routing group? I call this a routing group, right? Like where it’s like collect Yeah. Things.

[00:36:58] David Singleton: So two things, to [00:37:00] some extent, dreamer does make its own tools in that agents appear to the system as tools.

[00:37:05] So they can be, they can be used to accomplish things. So you can build an agent that is essentially a tool. Yeah. Um, and it it,

[00:37:12] swyx: which is to me very useful for reuse.

[00:37:14] David Singleton: Right.

[00:37:14] swyx: Right. Exactly. ‘cause I, I like, this is the way I like it. Now my next five apps, I don’t want to do this whole series of back and forth again.

[00:37:20] David Singleton: Right.

[00:37:21] swyx: Yeah.

[00:37:21] David Singleton: Um. Then at the tool layer of the system, it’s open to anyone. So it’s actually quite powerful and flexible. So if you wanted to add a tool, which was, uh, imagine that you were training your own foundation model, Swyx. That might be fun. And imagine you wanted people to be able to play with, I don’t know, maybe you make like, you know, nano chat or whatever and you want to Yeah.

[00:37:42] Let people play with your own nano chat and see how I change themselves.

[00:37:44] swyx: Now.

[00:37:45] David Singleton: You could, you could publish a tool that is Nano Chat and it nano image generation behind a tool, and it could be your own writer if you wanted to. I see. And honestly, if that’s the kind of thing that gets you excited as a builder, please come and do it.

[00:37:57] Like we, we really are [00:38:00] believers in this idea that we aren’t going to figure out every single detail ourselves. We’re gonna make sure it’s a safe and fun place to build this stuff, but we’re really open to these ideas coming from other people. Um, and so I’d like nothing more than you come in and build a tool that does some of that cool stuff that you, that you have in mind.

[00:38:15] swyx: Yeah. Awesome.

[00:38:16] David Singleton: And just as a reminder, if you’d like to do that, the way to find the links is dreamer.com/latent space. Um, and for a limited time on that page, um, anyone who’s listening to this podcast will also get directly off of our wait list. Uh, it’s quite long right now. We are working hard to bring Zika.

[00:38:32] Wait, so skip the wait list.

[00:38:33] swyx: You know, I think, I think that’s fantastic. I, I think it’s, it is really sort of probuild way to do it. I wanted to jump back to the, the bar. Yeah. You know, you know, I get excited about this.

[00:38:41] David Singleton: Yes. Okay. Let’s set it back in there.

[00:38:43] swyx: Like, let’s, you know, this is the engineer podcast that’s get

[00:38:46] David Singleton: Yeah.

[00:38:46] swyx: As technical as you can.

[00:38:47] David Singleton: Yeah.

[00:38:47] swyx: On everything you’ve built, like have a show off.

[00:38:50] David Singleton: Yeah. Okay.

[00:38:51] Under The Hood Debugging

[00:38:51] David Singleton: So let’s go wild in the aisles in the Asian studio. So as you can see, over on the left here is a conversation with the sidekick where you ask it what to do and it will explain in English that anyone can understand what’s going on.

[00:39:03] But, um, if you want to pull back the covers and look under the hood, um, if you’re, uh, an engineer like me, then we have this, uh, this kind of debug drawer at the bottom. So you can see the full build logs here, but you can actually also dig in and see the files and prompts that have been generated. Uh, you can upload files from your computer in static files.

[00:39:24] Um,

[00:39:24] swyx: very important,

[00:39:25] David Singleton: uh, indeed. You can actually read the prompts that have been generated for you. We intentionally put an example in here just that you can see what the format looks like. And then, you know, we already looked at this one that was generated for this particular, um, app, but if you actually want to bring the code out of Dreamer and work on your own local machine, you can.

[00:39:45] So at the core of everything here is an SDK with a powerful command line interface and we built that first. It’s actually possible to build agents on Dreamer without talking to the sidekick. You can write code with your fingers on a keyboard if you want to. I know that’s very [00:40:00] antiquated, not, but actually this can be a lot of fun.

[00:40:02] So if you wanna pull it out onto your laptop, you can use our, our CLI and, uh, you can edit it in cursor or in cloud code. You know, you don’t have to use our sidekick. And the CLI actually has full access to the rest of the platform with you as the user. So, you know, obviously it is, uh, secure and privacy sensitive, and this is a way that, um, some of our most technical builders do build stuff on the platform.

[00:40:24] The really cool thing is the side cake. When it’s in coding mode, it uses exactly the same CLI. So the way it. Build stuff on Dreamer is using the same tools that you might as an engineer. Um, and that’s actually a very powerful abstraction because it turns out that the right way to give a lot of context to agents to use CLIs is to write great documentation.

[00:40:46] Make sure that all of the things that you could do are actually possible. And guess what? That makes it a delightful developer experience for real heroes as well.

[00:40:53] swyx: Yeah. So that’s pretty cool. We’ve been telling developers to do this and they ignore this until now they have to for content.

[00:40:58] David Singleton: I, I’ve been saying this for a [00:41:00] long time.

[00:41:00] Uh, we actually Stripe docs.

[00:41:02] swyx: I mean, come on. Absolutely. Come on.

[00:41:03] David Singleton: Absolutely. But actually, I was chatting with folks at Stripe last week and saying, Hey, you gotta make the Stripe CLI actually tell agents what they can do on Stripe because that way they’re gonna use more stuff on Stripe. I think this is a real trend for the entire industry.

[00:41:16] swyx: Yeah.

[00:41:16] David Singleton: So we, we’ve been doing that.

[00:41:17] swyx: To me, this, this download and, uh, GI push mm-hmm. Everything is complete confidence in that you’re not hacking it. Right. Because there’s other, let’s call them AI builder platforms that impose their stack on you and if you, if you, and so therefore they don’t allow you to do this because they cannot.

[00:41:34] Right. ‘cause they, they impose some degrees of freedom, uh, restrictions so that they can get it to work. Yours is a fully general like VM running the full code. Correct. Do whatever you want. Correct. Any language you want. Correct. Yeah.

[00:41:46] David Singleton: Correct. Well, in terms of language, if you use the SDK, you could build stuff in other languages.

[00:41:51] We’ve actually found that TypeScript is the best language for building these experiences. Yes. Because it’s strongly tight. So you find out at compile time if you’ve made mistakes [00:42:00] and there’s nothing better than getting in. A coding agent in a loop where it can see its mistakes and ask them. So TypeScript is the language that everything gets built in by default here.

[00:42:08] swyx: Did And did you see that TypeScript overtook Python? I did. I did. Yeah.

[00:42:12] David Singleton: And for what it’s worth, when we started the company, we started writing stuff in Python, and I love Python. Um, if I do, uh, a vendor code, I always write it in Python. It’s my favorite language as a developer with my fingers on the keyboard.

[00:42:23] Um, but TypeScript is an amazing language for AI because there’s tons of training data in the models, um, and it’s strongly tight. And actually at the company we built most of the stack in TypeScript, and we have this amazing property, which is, we have type safety all the way from the database to the front end.

[00:42:40] And there’s nothing better for working with coding agents than being able to have them check their correctness, compile time. So the same ideas behind building the company’s code base, we’ve put into the agent SDK here as well.

[00:42:51] swyx: Yeah. Do you know if you’d use one of those tools, like Prisma or whatever, or is it Tool Lab for you?

[00:42:55] David Singleton: We, we actually have crafted most of our own tools. Um. For [00:43:00] instance, we had LLM Driven Code Review, uh, before the thing that got published from philanthropic this week. You know, we, we’ve been doing this stuff, uh, on our own bat

[00:43:07] swyx: email, we’ll pay $25 per review.

[00:43:09] David Singleton: We, we pay a lot less than that. However, I hear that those reviews are excellent and possibly worth $25.

[00:43:14] swyx: Yeah. You know, it’s an option. Right. It’s good, good to have it.

[00:43:17] David Singleton: Just to give you a tour of some other stuff here. So, um, I can also see all the versions. Yeah. Um, this is not gi, this is not gi, this is built into dreamer. I can see all the versions that have been pushed before. Why is it

[00:43:27] swyx: not gi?

[00:43:28] David Singleton: It’s not gi because we can make it work more efficiently than Git.

[00:43:32] And we actually, we do some work behind the scenes to kind of understand what’s in each of these versions. Yeah. Um,

[00:43:37] swyx: so one of the things I’m pursuing, and I have a lot of thesis, right? Mm-hmm. One of the thesis is like, does GI go away? Does GitHub go away? And like, what, what is the active reinvent

[00:43:46] David Singleton: you for, for what it’s worth to some extent.

[00:43:48] And anything you build, there’s a lot of path dependency. If we started over, we might make this gi There’s, uh, you know, within the company we use, uh. For our, you know, platform source code. And we like it and it [00:44:00] works well with coding agents as well. The very first versions of this, we wanted to be able to make it possible for the sidekick to manipulate it easily.

[00:44:06] Um, and this, this was an expedient way to do it.

[00:44:08] swyx: Yeah.

[00:44:08] Workflows Logs And Databases

[00:44:08] David Singleton: Um, you can also see all the activity that has happened in the workflows that you build. A lot of agents, you’ll build on Dreamer, do things in the background, so they run on triggers. These are stimuli from the outside to kick them off, and this is a nice way to see all of the things that might have kicked off your agent.

[00:44:24] You know, you can have an agent that kicks off on a webhook, so you can plug it into external systems. You can have an agent that runs when you receive certain emails that match filters, including LLM filters. And so here you can see, oh, when did it run? What did it do? You know, if I open up one of these guide me prompts or guide me, uh, events.

[00:44:41] Oh my can see God. Well, I told you it was calling an LLM for every one of those time slots. Here’s all of the LLM calls, here’s the actual prompts.

[00:44:49] swyx: And you don’t mind exposing all of this, right?

[00:44:51] David Singleton: No. We want builders to see what’s going on under the hood. It’s haiku to,

[00:44:53] swyx: okay. Yeah. So,

[00:44:54] David Singleton: okay. Right now that one was haiku.

[00:44:56] Like I said, we work with all the models and sidekick will actually pick the best one [00:45:00] for the job. And you saw that was pretty high quality and pretty fast. So Haiku four five is the one that it picked for that job. Exactly. Uh, we also have logs, as I mentioned, there’s a database spun up on demand for every, uh, agent.

[00:45:12] You don’t have to go and figure out how to do your own hosting. This is a SQL Light. This is a SQL Light database. Yeah. Um, it’s a multi-user SQL light database. And then, uh, but, but each one is you, you get a database that is unique to this agent. But then if you share the agent with multiple people, we take care of like who are the owners in each row?

[00:45:31] And all of that stuff is just there outta the box. Um,

[00:45:34] swyx: and again, in-house?

[00:45:35] David Singleton: In-house.

[00:45:36] swyx: Oh my God.

[00:45:37] David Singleton: Yeah. Um, well we do work with a bunch of infrastructure providers, but the technology for how to manipulate this is in-house. Fun fact. We actually did a lot of our own infrastructure development early on at the company and realized we need to spend our energy in the stuff that we’re uniquely doing in the world.

[00:45:53] So we’re very delighted to partner with a bunch of great designer and some of this stuff. And then finally, um, I mentioned that agentic apps agents [00:46:00] expose all of their internals to the system so the psychic can manipulate them and use them just like a user can. So you can see how it’s decided to break this problem up into functions.

[00:46:09] Some of the functions, the ones with the little I here are exported. That means that there’s probably the visible from outside. Exactly. And others are internal. And if you want to, you can dig right in here and call individual functions and see what happens. But mostly. You don’t need to think about that at all.

[00:46:24] Yeah. Uh, you can keep that little drawer closed and you can talk to your sidekick and build really powerful and enchanting experiences.

[00:46:30] swyx: Yeah. I mean, to me, like showing this gives the engineer a complete mental model of what you’ve done and what you can do with it. Yeah. For example, the first thing I, I, I look for.

[00:46:39] A mental checklist of things, right? Like is off in the database, off looks like it’s not right. So that’s a separate layer. That’s probably me means it’s hard to do multi-user apps on the same app, right?

[00:46:50] David Singleton: So you actually, we’ve solved that. So, um, see, yes, the platform builds in off, so you as a user sign into the platform, if you’re using an [00:47:00] agent that was published by someone else, then your identity is, is kind of taken care of by the system.

[00:47:05] And when you query the database, you’re gonna get the stuff that is for you. Unless the builder specifically said, this is public data that everyone should see. So they, they actually get a chance to think about that. And again, sidekick can guide you through building, uh, agents and apps that work that way.

[00:47:19] So you’re right, that’s another thing that people have to think about when they’re trying to figure out how to build software experiences on Dreamer. You, it’s built in. You talk to the sidekick as if it were a human being about what you want and that’s what you get. So, you know, my, my Big Sky app that I just showed you that was designed for multiple people to use it.

[00:47:38] And of course the things that we were putting in as expenses were supposed to be visible to everybody, and I just told the sidekick that’s the way I wanted it. Uh, but by default, if I built an app like that, the data from each user would not been visible to the others.

[00:47:49] swyx: Yeah. Yeah. Uh, this is, I presume this is a mood question, but basically you’ve had to build your own coding agent, right?

[00:47:55] Which is sidekick slash whatever is in Inside Psychic. Obviously there’s a lot of [00:48:00] people with a lot of desire for cloud code and Code X and attachment to it. Mm-hmm. I know under the hood data basically reduced to a loop, but like, would you let people use cloud coding and Code X or is the harness too specialized?

[00:48:12] David Singleton: Yeah. If you, if you want to use, um, cloud code and Code X, then you go down here. Yeah. Hit get the S St K. And we even say this right here, edits your heart’s content Z cursor code.

[00:48:22] swyx: Like people want to use it inside of Ick, right? Yeah. They want to switch the engine.

[00:48:26] David Singleton: Yeah.

[00:48:26] swyx: That’s the coding engine.

[00:48:27] David Singleton: Yeah. We are not doing that right now.

[00:48:29] Um, you know, again, the goal really is abstract the complexity. Yeah. Um, because the real target for. Building agentic apps is folks who can’t do this already today. I can’t tell you how many users in our community I’ve spoken to who are like Dreamer has changed my life because I used to have all these ideas.

[00:48:50] If only I could find an engineer to help me implement them, I’d be able to get them done. They’re free, and now I can talk to my sidekick and, and get it built. I think that’s like really how we think [00:49:00] about the people that should get a ton of value and fun, um, out of the platform. And so they’re not asking to be able to plug in their their own, you know, coding agent.

[00:49:11] And for those folks, the opportunity is massive. If you’ve never been able to do stuff in code, now you can build stuff for you, for your friends, for your family, for your coworkers. And also there’s a huge opportunity for folks who do build stuff in code to actually contribute to this ecosystem. So that’s how we think about it.

[00:49:28] swyx: Yeah. Amazing.

[00:49:28] Personalization And Memory

[00:49:28] swyx: That’s most of what I wanted to cover Dreamer wise. I think personalization and memory yeah. Is probably like the single most important job of, uh, of the os. Maybe we could talk about that and then I’ll, I wanted to zoom out on company building stuff.

[00:49:40] David Singleton: Yeah, yeah. Sounds good.

[00:49:41] swyx: Yeah. So how do you handle memory?

[00:49:43] What, yeah, what have you found? What have you tried and failed?

[00:49:45] David Singleton: Yeah. Okay. So, uh, first of all, at the core of dreamer is the sidekick. The sidekick gets to know you and it builds up a memory about you over time, and that turns out to be very important. So Dreamer, that’s your moat. That’s Dreamer gets better the more you use it.[00:50:00]

[00:50:00] For instance, a lot of agents in the platform, when you start using them, the first thing that they’ll show you, here’s what I think is relevant to you for this particular use case. Uh, a very popular kind of agent on Dreamer is a weekend activity planner. So, um,

[00:50:14] swyx: like, just tell me what to do.

[00:50:15] David Singleton: Well, tell me what to do, especially if I’ve got kids, right?

[00:50:17] So I have two kids and a dog, and my wife and I often spend a lot of time trying to figure out what are we gonna do with the crew this weekend. And, you know, we have interests that are very consistent. It actually can take a ton of work during the week to figure this out. So there is an agent on Dreamer called Weekend Activity Planner, and it helps me find things to do with, with the family of the weekend.

[00:50:39] In fact, this morning I got a message from my weekend activity planner telling me about the St. Patrick’s Day parade on Saturday. Oh, at Civic Center. I’m Irish. My kids are technically Irish as well. I mean, they, they, they have multiple citizenships, but you know, they’re, they’re Irish. Um, what a better thing to do than take them to the St.

[00:50:56] Patrick’s Day parade. Why did that get recommended to me? Because in the [00:51:00] profile that we can, activity Planner knows about me. It knows that I’m Irish, right? So all of that comes from the memory that Psychic builds up about me over time. We have invested in this a bunch. We will continue to invest in this more.

[00:51:11] We’ve tried actually many different techniques. As, you know, the, the kind of, um, cutting edge of a agentic memory has changed over time. You know, very early on we were putting lots of facts into a vector database and, uh, and doing embeddings and pulling them back out, um, using, you know, reverse lookup of embeddings rag that actually worked, but turned out to be much more complexity than was actually required.

[00:51:33] So, you know, today we’ve replaced it with a different system. Uh, I think we use a system that’s pretty similar to what you’ll find in lots of other products, but it’s an area that we’re actively, uh, investing in. Like, there’s, there’s. More than one person at the company specifically working on memory. And so expect us to just continue to make it better.

[00:51:50] swyx: Did you try knowledge graphs?

[00:51:51] David Singleton: We’ve tried knowledge graphs. The system that we have now is not a knowledge graph. Yeah. Um, but we’ve probably implemented most of the papers you’ve seen out there on agent [00:52:00] memory and the current system is working pretty well.

[00:52:02] swyx: Yeah. Excellent. Zooming out just on the company stuff.

[00:52:06] Mm-hmm. Um, uh, this is your first time in the CEO seat. Correct. You were CTO before. Correct. What’s different?

[00:52:11] David Singleton: Yeah. The difference between being a CEO and A CTO really is just. Like making sure you’re looking across everything. So, um, I have run products before, so for instance, Android wear, you’re basically a CEO

[00:52:25] swyx: of

[00:52:25] David Singleton: that product.

[00:52:26] I, I, I was running that as a general manager.

[00:52:28] swyx: Yeah.

[00:52:29] David Singleton: However, when you do it for your own company and the buck truly stops with you, it definitely kind of raises the temperature a little bit. Um, but it’s been a lot of fun for me to think about a lot of go to market topics. Um, I spend a lot of my time these days meeting users, uh, talking to folks about what they could do on the platform, being very active on X and LinkedIn, uh, which by the way is a huge pleasure.

[00:52:51] It is so much fun to be able to engage with users and potential users directly and understand what they would like to do. Um, and that’s the biggest difference [00:53:00] between this role and being the CTO, um, of, uh, of a company. At the same time, I am someone who always likes to look for why are we doing this?

[00:53:10] Who are the people that. Benefit from it. So, you know, even as A-C-T-O-I was always paying a lot of attention to topics across the company. So I feel very grateful for all I learned in my previous roles that kind of got me ready to, to do this at this kind of scale.

[00:53:24] swyx: Yeah.

[00:53:24] Tiny Teams Hiring And Taste

[00:53:24] swyx: To me this is like the natural lead into when I went into your office.

[00:53:27] Yeah. It’s surprisingly small.

[00:53:28] David Singleton: Yes.

[00:53:29] swyx: So, and I have a, another thesis I’m pursuing for latent space, which is the emergence of tiny teams. Yeah. Where, uh, you know, the, the classic sort of image is that teams with more millions in revenue than employees, right? Yeah. So you, that’s revenue efficiency definition.

[00:53:43] But I do think as a CEO, you are going to run a smaller team than you used to.

[00:53:46] David Singleton: Yeah. So I believe very strongly in the power of small teams. So the more people you add to a team, the more communication overhead there is. And it doesn’t even grow linearly. If you think about it, the more people you add, everyone cares [00:54:00] about getting to know everybody else.

[00:54:01] And sharing what they’re doing with everybody else. And that’s great. I’m not saying they shouldn’t, right? The very, like, I wanna work in teams that are fun, where people are talking to each other and, and sharing ideas and so forth. But, you know, there’s just a kind of gravitational weight that comes from larger and larger teams.

[00:54:16] So just inherently large organizations are less nimble than small ones. And if you run a large organization, you have to keep thinking about how do I kinda like prune things so that it can act like a small team. So a dreamer, the, the core team that built everything I just showed you was, was honestly about six people.

[00:54:34] Uh, we’re larger than I we’re about 17 people at the company now because as, but

[00:54:38] swyx: still, uh, for everything you just showed,

[00:54:40] David Singleton: it’s, it’s still a small team, which is great. Very, very high talent density team. We’ve been very, very careful and kind of obsessed as we grew to make sure that everyone that’s joining the company is joining a team that they’re gonna get a lot of, uh, learning out of, but also they’re actually going to kind of.

[00:54:57] Help everyone else a lot as well. There’s something very [00:55:00] special about that too. You know, every single person at our company I would be delighted to do any project with at any time because, uh, they’re just all great. And, you know, the smaller you keep the team, the easier it is to make sure that, that that talent density is there as well.

[00:55:14] Of course, it’s a real luxury to be building a company. We started this company in late 24, but it’s a real luxury to be building a company today because we can build with agents. So we’re using coding agents.

[00:55:26] swyx: Yeah,

[00:55:26] David Singleton: we’re using Dreamer marketing agents. All of our operations. We’re looking at how we can, can actually accelerate what we’re doing, uh, using our own tools.

[00:55:36] swyx: Um, any, actually any agents that you don’t build that you wanna shout out? Just that, that you love?

[00:55:41] David Singleton: Yeah. Is it

[00:55:41] swyx: other people’s

[00:55:42] David Singleton: agents that we built for the

[00:55:43] swyx: company? No, no, no. Other people’s, uh, stuff like you shout out granola.

[00:55:46] David Singleton: Yeah. So I showed you Attain finance. Uh, attain Finance has an agent as well, which like helps you manage your money.

[00:55:53] I find this really amazing. So, um, I always have this like lingering feeling that I’ve got a whole bunch of [00:56:00] subscriptions that if I just had a bit of time to go across them, I could, you know, figure out how to consolidate them. And the person who built Attain Finance doesn’t work at our company. What they were part of the early Alpha group.

[00:56:10] So they gotta kind of look at how all this stuff works pretty early on. And they built this really amazing experience that actually helps you. Like, save a lot of money because it will kind of help you analyze your purchases. It’s almost like a kind of a financial fitness coach. He’s called Andrew, uh, who, who built it.

[00:56:26] He came and showed it to us and the first thing it did was it recommended that he should buy fewer burritos. And, uh, he was like, it’s true. Like that is actually how I could save the most money. So, uh, that’s a, that’s a pretty cool example.

[00:56:38] swyx: Uh, I noticed he was first. Because he’s order alphabetical order.

[00:56:43] So I’m, I’m wondering how come there are no like Avar? Uh,

[00:56:46] David Singleton: yeah. Well if you’re a builder right there and you’re wondering how do I seo o myself on the Dreamer platform, Swyx suggest you name your tool Avar. In all seriousness though, those are the tools I have connected. So they’re in alphabetical order.

[00:56:58] If you haven’t yet connected them, we actually [00:57:00] kind of put them in the right order for you. So if Sidekick understands you and puts in the right order, uh, but I’d say a arc is gonna come before, uh, anything else,

[00:57:06] swyx: right? Yeah, exactly. Um, and, and then I, I think how has hiring changed? Yeah. You’ve hired plenty of self engineers in your life.

[00:57:14] David Singleton: Mm-hmm.

[00:57:14] swyx: I assume something’s changed.

[00:57:15] David Singleton: Yeah, absolutely. So one of the main things that I look for now when hiring engineers is. How well do you work with coding agents? Our team actually is quite experienced a good number. Everyone at Dreamer, other than, well, I guess I write a lot of code too. Everyone’s an ic, an individual contributor.

[00:57:32] Many of the folks that work on the team have previously been managers. And it turns out being an engineering manager, as long as you stay very close to the code and are able to continue to craft it yourself, is actually a great skill profile for being able to make agents work for you and for your team in this, uh, in, in this age.

[00:57:50] And so that’s definitely something that we look for quite intently when hiring engineers. And, um, we still have folks write some code like with their fingers. It’s just important to know [00:58:00] that the kind of core of the craft is there. But the vast majority of what we spend time doing is building quite significant and elaborate stuff together in a fun, collaborative environment with coding agents.

[00:58:09] swyx: Right.

[00:58:09] David Singleton: Um,

[00:58:10] swyx: so what, what is the interview loop like? Sit there with Codex, do something.

[00:58:13] David Singleton: Yeah, I mean, our interview loop is one a coding. Screen to make sure that the, the base is there. And then we actually do a couple of short projects, uh, with an engineer on our team and whoever is thinking about joining, where we’ll actually put out a very fully formed product idea, we’ll riff on it together and make sure that we can test product sense a little bit and we’ll actually try to build the whole thing with x or cloud code or whatever, uh, whatever the person is most familiar with.

[00:58:39] Um, and watching how someone thinks about prompting the agents, what they do while the agent is working. ‘cause you know, you can actually, this is a kind of interesting, uh, dynamic in the industry. Anytime I’m working on code these days, I always have more than one agent going at the same time because while one agent is going and reviewing the output of the next one, and if you [00:59:00] get them in a nice round robin, you can be very, very productive.

[00:59:02] You can also chain agents together. You can have one agent producing code, another agent reviewing it. And actually just seeing how folks have adapted their workflow, um, is a big part of what we’re we’re looking for in our interview process.

[00:59:13] swyx: Amazing. I guess last question, but also open to you to bring up any topics that I haven’t touched on, have you wanted LLMs to do that they still cannot do today?

[00:59:23] David Singleton: That’s a great question. Um, and it’s amazing ‘cause the capabilities of LLM just, just advanced so quickly. You know, if you’d asked me a year ago, I would’ve said, well, you know, music generation, I, I like music. Um, and Suno is amazing by the way. And, but previous generations i’d, yeah, I can kind of tell that that’s AI generated today.

[00:59:42] I listened to the latest tracks made by Suno. I’m like, that’s, that’s pretty impressive. If we went back six months, I’d be asking for better image generation. The latest nano banana, uh, which by the way is a tool on the platform that you can use on Dreamer is producing spectacular infographics.

[00:59:58] Spectacular [01:00:00] painterly images when I ask for those as well. So, so that’s quite impressive. I still think I, so I think as we go forward into the future, there is still a lot of room for human creativity and so that’s also maybe where I’m going to have to say that LLMs are most lacking. So I think that when you think about building software, the thing that’s really important and that we all need to bring is taste.

[01:00:24] Mm-hmm. Right? You have to like actually truly understand people, their motivations. How do I build something that’s really delightful? So, you know, we had to do a lot of work on Dreamer to make it possible for the experiences that we build to not look like AI generic slop.

[01:00:43] swyx: Right? We go,

[01:00:44] David Singleton: um. And we’ve done that by putting a lot of our own taste into the templates and the prompts and the, the harness.

[01:00:52] Um, so I hope you have fun playing with it. I, I, I think Dreamer today generates experiences that don’t feel super generic, but that’s a ton of [01:01:00] work, right? The LMS do not do that by default. And in fact, I, if I see a, if you ask for a simple like to-do list app or something, uh, built by the models, I can tell which model built it just by kind of how it looks.

[01:01:10] So, um, taste, creativity, sense of individuality is still something that I think the LLMs are not producing out of the box. And I think that’s gonna be an interesting frontier. What do you think?

[01:01:21] swyx: Usually that’s, this is by, uh, from builder to researcher question. ‘cause uh, we do have researchers listening.

[01:01:27] Yeah. And I’m just like, well, that’s it. But like soft taste for me please is, is like a very broad topic. Uh, what do I think? I mean, I agree. I just think that it’s too big of a topic to break down. Mm-hmm. Particularly because there’s a lot of, I’ll know it when I see it type, uh, eval, which is unverifiable for, for researchers to do so.

[01:01:45] David Singleton: Yeah, I mean I, I do talk to researchers quite often and, uh, we talk about this topic and I think most people agree

[01:01:51] swyx: uhhuh

[01:01:52] David Singleton: that, you know, one of the great things about building models to generate code was just, you know, it’s so verifiable. So Yeah. Um, you know, it’s [01:02:00] very tractable and they agree that the next problem is how do you kind of step up that hierarchy of needs and get into these taste questions.

[01:02:08] And quantifying taste is hard, but I’m actually kind of excited that some people are gonna start doing this. And you know, that’s when I think that some of the really iconic companies in the world will start to become places where, you know, folks really try to like. Debug and understand the creative process.

[01:02:23] And I get pretty excited about that.

[01:02:25] swyx: Yeah. Uh, I, I think we are slowly uncovering what intelligence really means and, and the, the benchmarks that we adopt and then abandon because they’re solved is, is basically us evolving the machine intelligence in the way that we, the different way than we evolved, but we are slowly understanding what it means to be intelligent.

[01:02:44] Right. And, uh, and it’s, it’s interesting. I wonder how they suppress us in the future, but like, we’re not even there yet. We’re just like, get, get it to a place where we like what we get. Mm-hmm. From the machinist sometimes. You know, it used to be 30%, now it’s like 95%, but still there’s that 5%. [01:03:00] That’s right.

[01:03:00] Yeah. Any other topics we should have touched on?

[01:03:02] David Singleton: No, I think we’ve covered everything, but I wanna remind everyone,

[01:03:06] swyx: ct

[01:03:06] David Singleton: dreamer.com/latent space.

[01:03:09] swyx: Yes. No, it’s a, it’s a very good deal. I mean, like, come on. Like, yeah. So thank you for offering that.

[01:03:14] David Singleton: Cool. Well Swyx, thank you so much. This was fun.

[01:03:16] swyx: Yeah, thank you.

[01:03:17] Uh, we, we’ll get Alejandro to put like flashing neon signs on the, on the YouTube. Cool. Wonderful. Um, alright. Thanks. So my cool,

[01:03:23] David Singleton: awesome, thank you.

Claude Cowork came out of an accident.

Felix and the Anthropic team noticed something interesting with Claude Code: many users were using it primarily for all kinds of messy knowledge work instead of coding. Even technical builders would use it for lots of non-technical work.

Even more shocking, Claude cowork wrote itself. With a team of humans simply orchestrating multiple claude code instances, the tool was ready after a brief week and a half.

This isn’t Felix’s first rodeo with impactful and playful desktop apps. He’s helped ship the Slack desktop app and is a core maintainer of Electron the open-source software framework used for building cross-platform desktop applications, even putting Windows 95 into an Electron app that runs on macOS, Windows, and Linux.

In this episode, Felix joins us to unpack why execution has suddenly become cheap enough that teams can “just build all the candidates” and why the real frontier in AI products is no longer better chat, but trusted task execution.

He also shares why Anthropic is betting on local-first agent workflows, why skills may matter more than most people realize, and how the hardest questions ahead are about autonomy, safety, portability, and the changing shape of knowledge work itself.

We discuss

* Felix’s path: Slack desktop app, Electron, Windows 95 in JavaScript, and now building Claude Cowork at Anthropic

* What Claude Cowork actually is: a more user-friendly, VM-based version of Claude Code designed to bring agentic workflows to non-terminal-native users

* Why “user-friendly” does not mean “less powerful”: Cowork as a superset product, much like how VS Code initially looked simpler than Visual Studio but became more hackable and extensible

* Anthropic’s prototype-first culture: why Cowork was built in 10 days using many pre-existing internal pieces, and how internal prototypes shaped the final product

* Why execution is getting cheap: the shift from long memos, specs, and debate toward rapidly building multiple candidates and choosing based on reality instead of theory

* The local debate: why Felix thinks Silicon Valley is undervaluing the local computer, and why putting Claude “where you work” is often more powerful

* Why Claude gets its own computer: the VM as both a safety boundary and a capability unlock, letting Claude install tools, run scripts, and work more independently without constant approval

* Safety through sandboxing: why “approve every command” is not a real long-term UX, and how virtual machines create a middle ground between uselessly safe and dangerously autonomous

* How Cowork differs from Claude Code: coding evals vs. knowledge-work evals, different system-prompt tradeoffs, longer planning horizons, and heavier use of planning and clarification tools

* Why skills matter: simple markdown-based instructions as a lightweight abstraction layer for reusable workflows, personalized automation, and portable agent behavior

* Skills vs. MCPs: why Felix is increasingly interested in file-based, text-native interfaces that tell the model what to do, rather than forcing everything through rigid tool schemas

* The portability problem: why personal skills should move across agent products, and the unresolved tension between public reusable workflows and private user-specific context

* Real use cases already happening today: uploading videos, organizing files, handling taxes, managing calendars, debugging internal crashes, analyzing finances, and automating repetitive browser workflows

* Why AI products should work with your existing stack: Anthropic’s bias toward integrating with Chrome, Office, and existing workflows instead of rebuilding every app from scratch

* Computer use one year later: how much better it has gotten, why vision plus browser context is such a superpower, and why letting Claude see the thing it is working on changes everything

* Why many “AI verticals” may get compressed: specialized wrappers may matter in the short term, but better general models and stronger primitives could absorb a lot of narrow use cases

* The future of junior work: Felix’s concerns about entry-level roles, labor-market disruption, and whether AI can compress early-career learning into denser simulated experience

* Why Waterloo grads stand out: internships, shipping experience, and learning how real teams build products versus purely theoretical academic preparation

* The agentic future of the desktop: what it means for Claude to have its own computer, whether AI should act on your machine or a remote one, and how intimacy with personal data changes the product design space

* Why Electron still mattered: shipping Chromium as a controlled rendering stack, the limits of OS-native webviews, and why browser engines remain one of the great software abstractions

* Anthropic’s Labs mentality: wild internal experiments, half-broken future-looking prototypes, and the broader effort to move users from asking questions to delegating increasingly long and valuable tasks

* Why the endgame is not just more capability, but more independence: teaching users to trust AI with bigger scopes of work, for longer durations, with fewer interventions

Felix Rieseberg

* X: https://x.com/felixrieseberg

* LinkedIn: https://www.linkedin.com/in/felixrieseberg

* Website: https://felixrieseberg.com/

Anthropic

* Website: http://anthropic.com

Full Video Pod

Timestamps

00:00 — Cheap execution and building all the candidates00:44 — Intro in the new Kernel studio02:47 — What Claude Cowork is04:18 — Why user-friendly can be more powerful05:33 — How Anthropic built Cowork07:09 — Prototype-first product development08:00 — Why local computers still matter09:20 — Skills, primitives, and platform leverage12:13 — Cowork’s architecture: VM + Chrome + system prompt15:38 — Felix’s own bug-fixing Cowork workflows17:38 — Local-first agents20:16 — Evals, planning, and knowledge-work optimization23:14 — What Anthropic means by evals24:21 — Scaffolding, tools, and why skills matter27:44 — Demo: YouTube uploads and self-generated skills31:03 — Calendar automation and cleaning your desktop34:47 — Browser context and why DOM access matters37:47 — Skills portability and plugins44:36 — Which AI categories survive?46:19 — Junior jobs, simulated work, and labor disruption52:00 — Gradual takeoff vs big-bang takeoff53:42 — Finance, taxes, and enterprise verticals56:24 — Vision and the improvement in computer use57:31 — Why Claude writes its own scripts58:06 — Should Claude have its own computer?1:01:26 — Windows 95 in JavaScript1:03:19 — VM tradeoffs and sandbox design1:07:23 — Approval fatigue and safe delegation1:11:18 — The future of Cowork1:12:27 — What comes next for agentic knowledge work1:15:13 — Electron, Chromium, and desktop software lessons1:22:16 — Multiplayer agents and coworker-to-coworker workflows1:26:05 — Anthropic Labs and closing thoughts

Transcript

Alessio: Hey everyone. Welcome to the Latent Space Podcast, our first one in the new studio. This is Alessio, founder of Kernel Labs, and I’m joined by swyx, editor of Latent Space.

swyx: Yeah, so nice to be here. Thanks to, uh, TJ, Alessio, Allen helping to set everything up. It looks beautiful. We even have the logo outside.

Yeah, kind.

Felix: It’s like really nice, right? When you walk in here as a guest, you’re like, ah, this is a serious production. You’re like, feel it immediately.

swyx: Yeah. Felix, you’ve been, you’re, you’re currently a product manager of Cowork or,

Felix: uh, really Technic

swyx: Eng. Yeah. The, the identities are kind of vague member technical staff.

Felix: I know member staff is like, the official title will carry around forever.

swyx: Yeah. I basically kind of wanted, like we’ve been. Kinda obsessed. I, I’ve been using it a lot, even for managing latent space. Like, uh, cowork helps me upload videos and like title things and like edit and everything. It’s, it’s like really amazing.

Alessio: Cool. He said multiple times Cowork has said gi in the group track.

swyx: Yeah, yeah, yeah. So, so we have a second, uh, we have a second channel, uh, for latent space tv. Uh, and I, uh, and uh, we basically, this is our Discord meetup. Um, and I I, we have like Claude Coworks, it might be a GI, I don’t know if we, we have, uh, uploaded it yet, but one of the sessions was like a, like a Claude cowork thing.

Felix: I, you have to see, I would love to see it. Like, I’m so curious, like one of the most fun parts of my job is like constantly see the weird things people use Cowork for because it’s obviously like very hard for us to actually design for specific use cases we do. But like every single person who’s like most amazed is usually amazed about a thing that I didn’t even expect cowork would be good at.

Um, we have a new designer and it’s one of the first small tasks. I was like, Hey, we need like a new emoji for cowork for our internal stock. It’s like a pretty small thing. I like, can you please do it? And he drew an SVG and just gave it to coworker was like, can you animate this emoji? And now it has like this beautiful loopy animation.

Um, and I mean, I think obviously this goes down to like, it turns out you can do more things with code than you expected, but it, it’s like that kind of stuff that is really fun to me. So, long story short, I would love to see like, the kind of things you’re doing.

swyx: I’ll pull it up. I’ll pull it up.

Felix: Yeah. Yeah.

swyx: Uh, but before we get into it, I, I think always wanna start with like a top level. What is Claude Cowork for people who haven’t heard of it? Haven’t tried it out.

Felix: Okay. Uh, real quick, Claude Cowork is a user friendly version of Claude Code. So the way it basically works is we have Claude Code and for us, fairly impressive agent harness that over December we noticed more and more people are using either, even though they’re not technical, they, they’re not at home in the terminal or they are at home in the terminal, but they started using Claude Code for non-coding workloads, right?

Like managing expenses or like filling out receipts or organizing a knowledge base. Like there was a big obsidian moment that a lot of people liked and we wanted to capitalize on that, but also bring, bring this capability to people who are not terminal native and who might not know how to like brew and store something.

So cowork is Claude Code running in original machine with a little bit of padding, a little bit more guardrails, making it a little safer and a little bit more convenient for people who don’t wanna first open up the terminal when they go to work.

swyx: It’s interesting, uh, that is kind of. Pitch that way as a more user friendly thing because I always feel like it, it, to me, I I treat it as like why I’m familiar with Claude Code.

Like we, we did a Claude Code episode Yeah. A year ago. But this one is like even more power user tools ‘cause it, uh, it kind of integrates much better with like clotting Chrome and, uh, in all the, all the other tooling. But like, maybe, maybe that’s like a perception thing, right? Like

Felix: No, honestly, I don’t think you’re wrong.

This is like a, a thing I’ve been thinking a lot about for like the last two weeks. So,

swyx: but when they say user friendly, it’s like, oh, it’s the dumb down version. But no, actually this is the superset.

Felix: Yeah. Like, I think a similar thing happened, A similar thing happened to me about 10 years ago, like maybe 12 years ago when I was at Microsoft and we started working on, on Electron and like browser-based technologies and cross-platform stuff.

And one of the first use cases was Visual Studio Code, which used to be a website. And the initial narrative was, or Visual Studio Code is, is like a more user-friendly version of Visual Studio. But in a similar vein, I think there was some voices saying, oh, this is. For serious developers, like, we’re not gonna use this.

Right? For like anything. And I think in the end what happened is people have different stories about why Visual Studio Code became such a big thing. But my personal, my personal belief is that the Hackability and the extendability has like played a pretty big role, right? You can hook in Visual Studio Code that like almost any workload, it’s so easy to hack on, so easy to put extensions for it.

And I think cowork might be hitting a similar thing where it’s very easy to extend and it’s very easy to bring into your workflows. Uh, so the convenience I think is a bit of a, it’s obviously the thing we strive for as developers, but I think the way people find value in it then is by probably mapping it onto whatever they actually have to do in their job.

Alessio: So end of last year, you see the spike of like non-technical usage and clock code. What’s the design process to say we should make clock code work? Because I mean, you built it in only 10 days. Um, I’m sure there was some discussion before on whether it’s easier to use mean. You know, like making, making like a desktop GUI is obviously one way to do it, but like there’s a lot of nuance in the product.

Like maybe talk people through what was like the trigger of like, we should build a separate thing. We should not build like a different plot code thing. And then maybe some of the more interesting design decisions that maybe you didn’t take.

Felix: Yeah, I think philanthropic, we’ve been thinking about ways to move people who are comfortable with using Claude to answer questions and bring more of the power of like this thing to now like, execute tasks for you.

I can like solve problems for you can like build things for you. How do we bring that capability to people who are currently mostly comfortable with like a like question answer paradigm within the chat. And we’ve had a lot of prototypes around that. Just going back as far as like easily a year and a half.

Like we had a lot of people working on that. Um, and internally philanthropic is a very prototype demo, first culture. We have a lot of like internal prototypes that don’t reach the public. What Cowork actually became is like we sort of picked the right pieces out of the many prototypes that we had.

Right. And that’s, that’s maybe also like, I think an important qualifier whenever people mention this like 10 day number. I do think it’s important to me to mention that within Double Scratch there was like a lot of stuff already happening, right? Like, and I think it’s important for people to remember that when you build a website, you use React, you use like a bunch of other things.

And this is like a similar scenario with like a lot of pieces we already had. Um, and in terms of decision path, I think we live in like an interesting new world where execution is actually quite cheap.

swyx: Mm-hmm.

Felix: So maybe, maybe what you would do That’s so crazy. The year. I know it’s wild.

swyx: You should be, ideas are cheap.

Execution is the hard part. I

Felix: know. And like the, we, we used to live in this world maybe where you would take a product manager and the product manager would go to a number of potential customers and in this like very low bandwidth way, would try to. Try to like tease out what are the problems they’re having, what are they willing to buy?

Um, and then maybe what can you build to like drive out that need and then you go back and you like draft a spec and you think about it and then like you make a design and you execute it. We internally philanthropic app, not pretty much closer to the point where we’re like, don’t even write a memo, just like build, like let’s build all the candidates very quickly.

Let’s just build all of them and then pick the best ones. I think the, the decision that is most impactful both for the product as well for the users right now is like the way we put value on your local computer. I think that’s a big decision point a lot of people have thought about. Should this thing, whatever it is, should it ultimately run into computer or should it run in the cloud?

‘cause they’re big trade offs, right?

Alessio: I guess like if we solve auth, it would be easy to do in the cloud. But I think like the fact that I can just download any file from anywhere and then put it and cowork there, it’s like a big unlock. Um, I mean it’s interesting you mentioned reusing certain pieces. I think this is something I’ve been thinking about even with Claude Code, right?

The price of like writing code is going to zero, blah, blah, blah. But it actually seems like the value of having some sort of platform substrate is like increasing because as you build these new things, you can kind of plug them together.

Felix: Yeah.

Alessio: So I almost feel like when people are saying, oh, the value of a lot of software is gonna zero because you can recreate it, to me it’s almost like the opposite.

It’s like having an existing platform to build on top of. It’s like even more valuable because you can kind of bolt things on.

Felix: Yeah.

Alessio: You have obviously mcps, you have skills, you have like obviously the models, which is a big part. All these things kind of come together. Do you feel like that’s a valid way to think about it, where people should invest even more in kind of like primitives.

To rebuild on or are you like recreating a lot of it each time because like things change and it’s easier to rewrite than reuse?

Felix: You know, I think, I think you’re right. I think you’re right that the holistic platform is really useful. And this is maybe a whole like a somewhat contrarian view to a lot of people in ai.

I actually don’t think that the future is going to be hyper personalized software down to the point where everyone is running their own version. Like, I actually think it’s going to be quite hard for all of us to have our own internal chat tool and like, if I wanna talk to you, like

swyx: how

Felix: is that gonna work, right?

In the, in the context of cowork and how we build it, I think it’s a bit of a combination. Like what the, the execution that gets cheap is not necessarily rebuilding all the primitives. I think our priori, there’s also not a lot of value in it. So for instance, my team did not think about rebuilding clock code.

We’re like very much started with the. The core thesis of this should be Claude Code.

Mm-hmm.

Felix: And then we’ll like build things on top of it. The part of the execution that gets a little cheaper is like, how do you take all of these Lego pieces and put them together in a way that makes sense for users?

It’s like actually valuable. You have so many different approaches now in terms of what kind of, what kind of things do you actually elevate to a primitive, do you strongly believe that all your products should be built by just combining primitive that the public also has available? Do you keep some things internal?

Um, and I think that’s still evolving, but I think what’s probably gonna go away is like, I’m not sure if it’s gonna fully go away, but I’m gonna say, I think for me personally, I will probably no longer try to come up with a really good product without testing up with people. This is not a new concept, but wherever you used to have to make costly decisions around, do we pick technology A or technology B, or do we like, um, build it this way, build it the other way.

I really strongly believe now you just build all of them and try them out with a small focus group and then whatever, whatever is better is what you go with. Right. And that, that is probably quite different even from how we maybe worked a year ago. Right. Like, I think, I think this happened very recently.

Alessio: Yeah. I started building something in on Electron since you’re here. Coincidence. Uh, but then Electron and like SQL Light are like, there’s like some issues that like between development and like, uh, building anyway. And I was like, let’s just rebuild the whole thing in Swift and just recreated the whole thing in Swift.

And it’s like, I. It’s done.

swyx: You know, I didn’t take any effort. I, I, I don’t even know Swift.

Alessio: Yeah, exactly. I was like, I’m the, I’m not reviewing it anyway, whatever. You can write in whatever language you pick, but the important stuff that I did was not write the electron bindings. Yeah. It was like the logic of what happens in the app, you know, and then the model is like, yeah, I can just recreate the same thing as with

swyx: Yeah.

I, I think you still want, especially for people who are doing like high performance software or like very complex software, uh, you still want like, some view of the architecture. Uh, but you can use markdown for that,

Felix: right? Yeah.

swyx: Uh, you don’t actually have to read the code again. I, I’m still like on a sort of like a definitional thing.

Um, can we build a good mental model of Claude Cowork? Um, this is what I have, right? Like you you said it’s like fundamentally cloud co. We don’t wanna touch it. There’s the cloud app, there’s clouding Chrome. I think you guys do something different in planning, but, uh, I’ve been talking with Tariq who is on the cloud co team, and you guys are, he’s like, no, we just exposed planning.

Maybe we can clarify like, what are the major pieces. That people should be aware. It goes into cowork, like,

Felix: okay, I think you basically have them. So really, um, you can, you can take planning more or less out. I think there’s a few things that are really valuable in cowork. Um, the virtual machine is probably the most powerful thing.

So we currently run like a, we currently run like a lightweight VM and we put clocked out into the vm and we do that for, for, um, a number of reasons. Safety and security is a big one, but even if you, even if you ignore for a second safety and security and you’re just like, okay, Yolo, I want this thing to do whatever.

It is quite powerful to give Claus on computer that is like generally a good idea. And in terms of architecture and UX and everything else that we’ve been working on, philanthropic, it often is quite useful for you to like anthropomorphize, um, clot aggressively and just be like, this is a person. What will you do if you give a, if you had a person, right?

Yeah. And the analogy I’ve given my dad this morning who is still like quite insistent on using chat even for like coding things, is if you were a developer and your employer told you that you don’t need a computer, they’re just gonna like, send you emails with a code and you send emails with code back like that, maybe work for Patrick Miles in the back, but that it’s not very effective.

Um, so what we can do with the VM is because it’s a, it’s a Linux system, Claude Code has more or less free reign to install whatever needs to install. It can install Python, it can install no js. We do have strict network ingress and egress controls. So you can still, as, as a user in like plain human language, make it clear to, to the entire system what you’re okay with and what you’re not okay with.

But at no point do we have to ask a real person, like a, like a person who might be in marketing or a lawyer. I’d have to go to a lawyer and be like, are you okay with me installing Homebrew?

Alessio: Yeah, yeah.

Felix: Right. Because the implications of the question and the answer are complex and nuanced and like, not, not easy to reason about.

This gives us a lot of distraction that makes Cloud very powerful. Now then around it, we, we do probably have a number of things that also keeps growing almost every single week that you’re probably noticing that make cowork maybe better for certain tasks than just cloud. Cloud on its own. Yeah. But most of those actually live in the system prompt.

They’re about like, what can we infer about the work that you do? What can we, what can we intru in the system prompt to make that more effective? It’s of course the like very tight integration with Cloud and Chrome. You’re noticing that a lot of people, especially as the models get better, a lot of people throw up their hands when it comes to MCP connectors in this area.

I’m not gonna, I’m not gonna go through like 25 M CCP connectors, click off everywhere and then like half of them don’t let me do the things anyway. So Cloud and Chrome is quite powerful because we can just talk to the cloud and Chrome sub agent and that will just do things for you.

swyx: Yeah, so, so one example right in MCPI, honestly, I think that the state of MCP is kind of, kind of.

Really hard to integrate. Um, I need to, I needed to add, uh, Figma MCP to the coding agent that I use.

Felix: Yeah.

swyx: Uh, and, but I didn’t wanna read the docs, so I just had caught to it. And it’s, it’s great at reading docs and the same, same way I had to set up like a Google Cloud, um, account for some project I was working on and get some API keys somewhere.

And Google Cloud is famously super hard to navigate, so I just didn’t wanna deal with any of it. I just used Claude Cowork

Felix: within the first week of developing on Core. This happened very, very quickly. Um, I caught myself by starting to use cowork for coding tasks, which is not ostensibly what we built it for, right?

We don’t need to. But I found myself, um, I found myself like on our internal, internal tool that we have for, to collect crashes and just like debugging information and I found myself sort like picking out the ones that I think we can easily fix versus the ones that might be like kernel corruption or something else on the operating system.

And I found myself sort of picking these out and then just telling Clark, go fix this bug. I was like, what am I doing here? Go one level up, tell a cowork, I want you to go to all these crash tools. I want you to find all the bugs that you think are fixable and not like an operating system crash. And then I want you to tell another cloud to like fix all of that.

Um, and that’s, that’s, that’s sort of another cloud,

swyx: just so it can spin up another instance or,

Felix: uh, it, currently what I do is, um, and this is a bit of a hack, but I tell it to use clockwork remote to which website itself? Yeah, that’s interesting. So you basically take, if you, if you imagine like a dashboard with like 20 bucks, you, this is remote control or clock or remote, or, sorry, I just wanted to confirm what, the way I’m using it is.

I have cowork running and I’m telling cowork, here’s where I normally go every morning to find the latest bugs. Go read the entire bug list, separate out which ones are fixable, which ones are, are fixable, and then for the fixable ones, four is this almost loop. For each bug, write a markdown file with a prompt.

And then for each markdown v, that is a prompt. Start of a cloud set. So natively Claude Code has

swyx: this concept of subagents. Mm-hmm. And this is basically a subagent, but you’re not using the subagent functionality.

Felix: I’m not using the subagent functionality. And the reason I’m not is because I’m firing that off as a Claude Code remote

swyx: task.

Felix: Yes. That’s kind of nice. ‘cause then I can just fire it off. I can go to my next meeting and in Claude Code remote. Now the work is happening.

swyx: Mm-hmm. Yeah. You, you see like you’re already starting to use the cloud over your local machine. And I think this is one of those things where like. Shouldn’t just everything just be cloud first, right?

Felix: Ah, this is such a good group. I’m like solely bad about this. I have so many thoughts about that. Okay. So I generally believe that Silicon Valley overall is undervaluing the local computer. And my default argument for that is always how come we’re all using MacBooks and not like an iPad or a Chromebook?

Um, that there is like still value in, in having a local machine. And now when I think about Clot, it’s this entity that is supposed to be very useful to you, like it tremendously useful to you. I think that entity needs to have access to all the same tools you have access to. Otherwise it’s gonna be hamstrung in like all these complex ways.

And there’s, there’s sort of two approaches we could take. We could say, okay, we’re gonna like one by one chip away at everything that is at your computer and move it into the cloud. That’s, that’s one way to do it. Um, and I think other products have taken that path. I personally, this is a very personal opinion, but I personally, for the amount of tools that I use.

Just don’t have the patience to give another tool like permissions to every single thing and keep those permissions up to date. The second thing that I’m still grappling with, and I don’t have a good answer for anyone just yet, but the second thing I’m still grappling with is what does it look like for someone to slurp up your entire work and put that in the cloud?

Like if I, just as an example, like if you could click a button and it just clone your entire computer into the cloud, is that something that you would want? I’m not totally convinced yet that all everyone will. Mm-hmm. And that is sort of like upstream of all the technical issues we’re gonna have. ‘cause like in general, I think the world is not ready for this kind of stuff.

Like, I’ll give you one quick example that would probably be very easy for us. So as a desktop app, we in theory with your permission, can do a lot of things on your computer, including reading your Chrome cookies. If we really want to do right, we could take your Chrome cookies, you would have to decrypt them for us.

We could put those on the cloud if we really felt like it. Pretty easy solution. That would be super cool. We could just be like, oh, we can do all your tasks in the cloud now. Um, a lot of websites, thanks, include it. If, if they see the same authentication from like two different locations, we’ll just lock down your account and now you have to go to the branch and be like, okay, I, I’m here with my passport.

You actually know that. Wow. Yeah. As tired as well are of the term agent for the age agent future, I think there’s a lot of stuff that sort of slowly needs to catch up and until that’s the case, the way I, as someone’s working on clock and make Cloud most effective is to like put it where you are working.

swyx: Anything else? I thought with our mental model, so like, basically like, uh, part of me also just want, like the more I understand how it works, the more I can use it to its full potential. Right?

Felix: Yeah.

swyx: And so what I’m get hearing from you is you told me to delete the planning thing. You’re not doing anything special on, on the, that’s only exclusive to Qua cowork.

Felix: We have some tricks for this sort of like change week over week. We eval cowork maybe against different use cases than he would evil clock code, right? If you think about it this way. Okay, so like clock code is our eval clock cowork. Yeah. So clock code is like quite optimized for coding tasks and we mostly value it whether or not we’re getting better or worse depending on how good it is at like a typical suite job.

And Clark Cowork on the other hand, we evaluate more against typical knowledge work, the kind of stuff he would find in finance or in like maybe a, like in like a legal office. Um, my personal use case is always like managing my things, like managing my personal mortgage or something like that, right? Or like wealth planning for me and my family.

Those are the kinds of use cases we eval, clock cowork on. And what you might be picking up on is like the subtle changes we make to the system. Prompt what we put in the system, prompt how we steer, clot with the tools we give it. Um, like either it’d be better in one or the other direction and whether there’s a trade off, try us exist a lot.

CLO code will be better of a code and Claude Cowork will be better. For non-coding tasks, will those gaps still exist in the next three generations of models? It’s like a little unclear to me though.

swyx: Yeah,

Felix: because right now these like hyper optimizations we make, I’m not sure for how long they’re still be relevant.

swyx: I think what I was referring to was also, it, it just, uh, it qualitatively felt different when I probably, it’s just all prompting and I’m reading too much into it, but like the, the fact that it comes out with like a nine step plan, I can edit the plan and give feedback and, and, and see it execute the plan.

Yeah. It felt more long range than in Claude Code, but maybe that already existed in Claude Code and you just build a nicer UI for it.

Felix: It’s kind of both. Um, like if the Clark Code people who build the planning functionalities would city, they probably say yes, we have all of those things in Clark code and they do.

Um, I think people tend to give cowork. Tasks that are maybe of longer time horizon, I thought is

swyx: so long. Yeah.

Felix: That’s like one thing, right? It’s just like that the, the chunk of work tends to be maybe a little bigger. And then the second thing is that because the work, when it gets longer, it gets a little bit more ambiguous.

We do tell co-work to make heavy use of the planning tool or to make heavy use of the ask user question tool, right? We do want it to come up with like. Different scenarios of, okay, tease out what the user actually wants. Don’t go off to work for like four hours and then come back with the wrong thing.

And you’re probably picking up on that.

swyx: Yeah.

Felix: Um, I wish I could tell you I like built this magical thing and it’s like, there’s some secret sauce,

swyx: but No, no, no. I mean, it’s, it’s just clarity is good that, you know, engineers just want to know. Yeah. They can, they can plan around it. And then I think also for me, um, I am realizing I have to switch to my, my other machine because this is a new machine that doesn’t have my session.

But, uh, yeah, the, the, the planning is really important for, for me to like approve or like to see whether it’s like, it’s right. The ask is, the question is so beautifully presented. I mean, it also, it also available in like cursor and, and in Claude Code. But like, I, I think like it’s so nice to see that it, like it’s kind of for me like to understand that it gets me, it gets what I want to do.

Felix: Yeah.

swyx: Yeah.

Felix: It probably very hard

swyx: just on the topical evals. Mm-hmm. When you say eval, I think people are very vague about what it means. Is it just like vibe testing or do you have like automated programmatic evals of Claude Cowork?

Felix: When we say eval, uh, what we really mean is that we essentially take the entire transcript, including all the tools that clot has available ultimately to it, and we then measure what are the outputs, depending on what we tweak, right?

So we do run that a lot. We use that in training. Um, we use that in, in like, if you sort of separate out post training from like the scaffolding around it. Cowork sort of exists in the scaffolding space, but obviously we also train on it a little bit. Um, so when we say eval, we mean given the certain transcript, what do the outputs look like?

Including the file outputs as well as like the actual token outputs, like the ones that you see in the chat window.

Alessio: I’m curious, um, how much of the failure modes are the model intelligence versus like the usage of the end tool to put the intelligence in? Like the well planning is like a good example, right?

It’s like one thing is to come up with a plan. The other thing is like make a nice spreadsheet. Yeah. That kind of runs you through the plan. Like how have you seen that? Well,

Felix: the thing that I grapple with a lot is that whatever scaffolding you come up with, I think we still have a bit of sort of like model overhang where the model is dramatically more capable than right.

Users end up using it for. And I think part of that is that we’re just not getting the model all the tools to do all the things that’s theory capable of, right? There’s like one thing, um, however, whenever you do build the scaffolding, I’m sort of wondering at what point, at what point will that scaffolding go away and like how much you invest in figuring out what the right scaffolding is.

It’s kind of up to, it’s a little bit of a bet. And one thing that I as an NJ quite enjoy is that like working in philanthropic and working at a frontier lab, I maybe have a little bit more insight into what’s coming, coming down the chute in terms of like, what’s the next model, what is the model capable of?

What is good at, what is it bad at? And I’m, I’m increasingly wondering, is the right thing for us to like really invest too much in sort of these like scaffolding corrections where the model might otherwise not misbehave, but just not do the thing that you want?

Alessio: Yeah.

Felix: Or is it to just like give it as many capabilities as possible, try to make those safe so there’s the worst case scenarios, likeno status might be otherwise.

And then just simply wait a second for the next model drop. I’m personally, currently more leaning into the ladder. I think we’re gonna see a lot of like applications and companies that do very impressive things with ai that in the short term might seem very effective ‘cause they’re very specialized to individual use cases.

But I think once models get better generalization and get better at like those specific use cases without being super guided on those, I’m not sure how long that’s gonna stick around. And you can kind of, kind of already see this in like skills and NCP servers, right? Mm-hmm. We’ve, we’ve already seen sort of this like slow shift from MCP service to skills.

And like, maybe a good example is Barry who made skills. He was initially hacking on something that honestly looked a lot, looked, looked a lot like what Cowork does today. It was sort of thinking about what if cowork, but for like people who don’t wanna build code. Mm-hmm. And, um, he too did that as a prototype inside the desktop app.

One of the first use cases we thought of were, okay, what, what are like coding like use cases that could really benefit from graphical interfaces and like from being a little separated from the actual underlying code. And everyone comes with the same answers. Data analysis,

Alessio: right?

Felix: Yeah. Or saying how many users do we have today?

How many, like, it’s always data analysis. And I think the thing that ultimately led to skills is that we wanted to connect this little prototype to our data warehouse and. The team very quickly discovered that like instead of building a custom tool for the thing to talk our data warehouse, they just like meet and embarked on follow like mm-hmm.

Dear Claude, if you want to get data, here’s the end point. Here’s what the API looks like. You’ll figure it out.

swyx: Ah.

Felix: And then it be hand over control. Yeah, yeah. Also just like maybe go one step up in the layer of abstractions, right. Just, yeah. Instead of, instead of telling the thing, here’s ACL I, please call the CLI, or here’s an MCP.

Please call this ECT shape. Just like this is the end point. If you wanna know something, if you post here, maybe you can do post sql. It’s gonna be okay. And that ended up being so effective that they started trying the same pattern of like just giving the model a markdown file that describes whatever it needs to do.

That the whole thing eventually became skills and we’re like. We should package this up. This is a good idea.

swyx: Yeah. Um, we’ve had Barry Mahesh, uh, on, on our conference and uh, he’s uh, definitely got a good idea there.

Felix: Yeah.

swyx: I wanted to show you the, how I’ve been using Claude Cowork.

Felix: Uh, this is was my favorite part.

swyx: This is this. So this is like me, uh, this is how we run the Discord. Uh, we literally, uh, at first I didn’t trust Cloud Core. This was my very first usage.

Felix: Okay.

swyx: Right. So then I was like, okay, I will just try to manually download from Zoom all my recordings and upload it to YouTube. Yeah. Because this is a very laborious process.

I got a click, click, click YouTube, um, isn’t super user friendly. Uh, and it just did it. And then I was like, actually, you know, even the download from Zoom part, I should also. Put into Claude Cowork, and then I did it right. Here’s a bunch of, and it starts compacting here, and it, and it, it starts to even be able to do things like look through the individual frames of the video to name the video so I can upload it auto automatically.

Oh, that is, and this replaces my job as a YouTuber. We will forever appreciate your creative Yes. You know, and so that’s great. Uh, but then by the way, it compacts and makes, makes like a new thing, right? So I, I don’t, I don’t have the initial, initial thing, but then I asked it to make its own skills so that it, so that something that’s repetitive and one-off and human guided becomes more automated and I can use the skills independently and reuse them.

Uh, and it obviously you can write skills and that goes into context and skills at the bottom here, which is, which is so nice. Um, so I have all these skills that, that I now sort of do on a weekly basis. Uh, I know you’ve released scheduled Coworks, which I haven’t done yet, but

Felix: course I should try them. I, I think this is like so wonderful and fun for me to see because.

One thing that is very fun for me about skills in particular is that they’re so easy to make. Like anyone can make a skill, like a text message, could be a skill, and they can be so hyper personalized to you. And this is like sort of the subtraction layer, right? Like, um, I, I’m just guessing, but I assume, heck, you are very good at your job.

You’re probably given this thing some guidance about how to do it, right? I,

swyx: I just said, wrap everything up into, into a skill, right?

Felix: Yeah.

swyx: And then, uh, and then I was like, actually, sometimes I might need to break, uh, things apart because some parts fail or some parts might be needed in individually. So I told it to split one skill into three skills.

So it’s like a skill splitting thing, and then there’s like a parent skill that just orchestrates all of them if I want to use that. You know, like, um, I think that’s, that’s like really good. Uh, and, and, uh, there’s, there’s one more part, which is the, uh, Google Chrome thing that I told you about.

Felix: Yeah.

swyx: Where I’m like, okay, you know, what’s better than uploading, using Claude Coworks to YouTube?

Like actually. Looking at the docs to like programmatically upload to YouTube and then putting that in a skill. And I’ve never done that before. I don’t want to deal with Google Cloud. Yeah. So Claude Cowork does it for me.

Felix: That is really cool.

swyx: So, so I, I just, I don’t care. I just, like, I do a thing. I don’t, it doesn’t really matter.

Felix: That is really cool. And then you’ve, I assume paired the skill just with the script that it’s built.

swyx: Yeah, no, I just update, update the skills.

Felix: Oh, that is beautiful. Yeah. That’s wonderful.

swyx: It’s kind of like a skill, like, uh, uh, basically I think like the way that people ease into Claude Cowork is like take a knowledge work task that you would normally be clicking around for and then, uh, try to turn, turn that, and then you do the, okay, well what if you went further?

Okay. And then when, if you went further, when, if you, and it sort of expand the scope of cowork as you gain trust with it and, and also teach it how to replace you.

Felix: Yeah. It’s like a little bit like playing factorial, but for your own life. Uh, like you say, you start really small.

swyx: Yeah.

Felix: You start automating something really tiny and like.

Once it clicks, you keep adding onto this like automation empire. Just like make your life easier and easier. My favorite skill has been, um, every single morning Kohlberg starts looking at my calendar and make sure that there’s conflicts because people tend to schedule a lot of meetings, sometimes last minute, sometimes miss it soft and painful.

And a lot of products have existed like that A lot. I’ve written in the custom prompt there. I haven’t made it a skill, um, honestly should.

swyx: Yeah.

Felix: But I’ve given it like pretty clear instructions about okay, here are some people, if they book over other meetings, I’m probably gonna go to their meeting. Like if Dario schedules a meeting.

swyx: Right.

Felix: Not try to reschedule down. Right. Um, and I think there’s some other rules in there about like what kind of meetings I care more about what kind of meetings I care less about. What is okay to like, maybe pun like when I want to be, when I want to be working, when I don’t want to be working. And it’s those really small things that I can think kind of click with people.

Right. When we launch co-work, I think one of the US races that went most viral on Twitter. X was clean up your desktop, which is stuff, because silly, that’s such a smart thing, right? Like you don’t need to model to clean up your desktop. Not really. Um,

swyx: like this, like clean up my desktop.

Felix: Yeah, exactly. Yeah.

swyx: I need to, I need to choose my desktop, right? I guess give it access to my desktop.

Felix: Yeah.

swyx: Okay. Uh, okay. This is very scary. Oh, we’ll do it.

Alessio: I did, I did it with my downloads folder. It was like, you have so many term sheets and there’s like eight copies of your rental lease for your office. I was like, all right.

Like, don’t yell at me.

Felix: It’s like, it’s not such a small task. And then like, I, I would never go out there and normally otherwise and tell people I’ve pulled a product. It can organize your folder. Right. Um, because it feels small. But I think to your point like,

swyx: oh, here’s, here’s the, here’s the ask user questions.

Felix: Yeah.

swyx: Uh,

Felix: beautiful. Right. Elite obvious junk. You probably shouldn’t click that.

Alessio: No.

Felix: If he’s not done right.

swyx: As long as it’s reversible, I don’t

Alessio: make up blend to,

swyx: yeah. Uh, yeah. No, I, I have a, I have a typical, everything is super messy folder. So, yes. I think this, this is super helpful. So this is a pretty simple task.

Mm-hmm. But I’ve, okay, here it is. Right. Here’s the progress. I don’t see this in, that’s why I’m like, this gotta be something different than, uh, than Claude Code, because I’m like, we

Felix: do. Yeah. That’s, we do system prompt that. We’re like, all right. We want you to think about like, this task Yeah. Methodology.

Yeah.

swyx: And then I can, I can, I can do like little suggestions for, for, for these things. It’s beautiful. Look at this. I, I can, I can like say like, oh, don’t do that. Don’t do this. It’s amazing.

Felix: I’m so happy. You like it. Um, I mean, the other way around, like we’re part of the Clark core team, if you would like this in Clark COVID.

swyx: Yeah. Yeah. Yeah. Uh, so, so yeah, I mean, uh, this is really good. Obviously I, I’m like kind of raving about it. Uh, you know, I have other things like sign up for pg e so if you can do phone calls for me, that’d be great. Um, I, I do, people

Felix: have done that. Obviously you can’t do that natively, but people have done that with like, various other providers.

swyx: Yeah. Uh, and then this is like signing up for the Figma MCP. Um, I, I really am trying to do like everything, um, data analysis as well. I do think, um, oh, design to code, uh, very, very good. Right? So like, here’s a Figma file, take it. And then this is where like a lot of other tasks is like knowledge work, like replace my manual clicking, but this is no, I would normally use Claude Code or uh, Claude Code for this, but because I perceive that you have better Chrome integration

Felix: mm-hmm.

swyx: I, I think you can actually do a better job of this. And I, this, this is one shot at my, uh, conference website.

Felix: That’s pretty cool. Like at some point I would love to like, hear how you feel about code. In the desktop apps, which is like I never use, which is the, the same team. Same team.

swyx: So I use the call code in terminal, which I, I perceive to be the default way of cloud coding.

Felix: So one thing this has,

swyx: sorry, I’m just like, I’m not

Felix: here, I’m not here. All products. Can I talk about other stuff? Like I, I’m not sure if people out there wanna like hear me advertise my stuff for like an hour. Please do that. Um, this thing is like a builtin browser, which is a thing a lot of products have said.

Yeah, it’s a builtin browser. And I think giving cloud eyes into like what you’re actually working on makes it so much more effective. And that’s probably what you’ve seen in cohort because it can see Chrome, it can like debug the dom, it can like see things. Um, that does make it more powerful.

swyx: Yeah. So, so I think, uh, my mental model was kind broken.

‘cause I only use this cowork because I thought it had a, a browser thing in it. But I understand that the Claude Code app. The app version of Claude Code does have a built-in browser. I’ve seen, I’ve seen this preview thing.

Felix: Yeah.

swyx: I just, I’ve never used it.

Felix: But in the end, in the end, you sort of have it by hard.

Yeah. You basically get the same thing. Right? Like the, the, the additional skill that you’re describing is chart is better if we can see what it’s working on. Right. That’s, that’s sort of like the summary here and like whether it’s using your Chrome

swyx: Yeah.

Felix: Or it’s just like making up its own little like browser.

It doesn’t really make a big difference because either way it’s gonna see what it’s working on and that just makes it much better. And then you don’t have to run QA for your cloud.

swyx: Why doesn’t it pick up my existing Claude Code sessions? ‘cause I, I mean, obviously I’ve used Claude Code, but Excellent question.

Um, don’t have a good answer other than like, we’re honest. Just haven’t Yeah. This is what the Open AI team does. Okay. Uh, cool. I I I don’t have other, like, I, I just, I, I do wanna expand people’s minds and also maybe show people if they haven’t really done it, but like, I, I think it’s very interesting how I sometimes use this more than I use, I mean, I use dia, right?

Yeah. Um, I, and I use, uh, I’ve used like all the other agentic browsers and philanthropic didn’t have to build an agentic browser because you just had Claude Cowork and that’s enough.

Felix: Yeah. I also think like maybe integrating with number of excellent browsers out there, it’s like currently on my personal priority list, a little higher than like trying to rebuild a browser from scratch.

Yeah. You know, never say never, but I think going back to this idea of like, we wanna plug this into an entire existing workflow, I think our goal is actually to not replace any of the applications we have in your computer. But instead of like, work really well within a new workflow,

Alessio: make the new one. Yeah.

Are, it seems that nowadays, especially on the browser, most of the innovation is like user ergonomics. It’s not really like the underlying browser engine. So I feel like to call it, it doesn’t really matter if it’s like the, uh, or Chrome or Alice, whatever.

Felix: Yeah. We wanna, we wanna meet you wherever you are.

Which is like, like obviously I would say that, but it’s also just generally true because I don’t wanna shrink my potential user base artificially by saying, okay, like, I’m gonna start building for the people who are willing to switch browsers.

Alessio: Right.

Felix: That’s such a, like, you know, like many lawsuits have been filed over who gets to review the browser and like a lot of money has switched hands over the question of like, which browser is default and which search engine is default within the browser.

Um, I just wanna build for, yeah, I wanna build for swyx essentially. Like, I wanna, I wanna, I wanna build for people who have a number of annoying tasks that they feel like. Maybe clock could do it. Could do it for them.

Alessio: Yeah. What do you think about skills portability? I think there’s been one thing, I use another thing called zo, which is kinda like a cloud computer plus agent.

And I have a skill to add visitors to the office. Yeah. So whenever somebody has to come in after hours, they need to check in downstairs. Um, but I wanna like text the thing, so it doesn’t really work in, in cowork, but now that skill is in the zone harness and it’s not in my cowork thing. And then if I make a change, it’s gotta, I gotta sync them.

How do you see that going? Like I see memory as like. Cloud personal, kinda like, I don’t necessarily want my memories to be cross thing.

Felix: Yeah.

Alessio: But I do want my skills to be cross agent that I use. I think with MTPs, people do the same thing. It’s like, oh, Mt. P Gateway. Mt P registry. I don’t really know if that’s like a business.

So I’m curious like if you’ve had any thoughts in the area.

Felix: I think for me, this is sort of where I go back to the really basic primitives for our skills are file-based instead of like this complicated thing that exists inside a place somewhere that is like super proprietary. I’m really leaning into the idea of like, it’s all just files and vultures, and that makes it very portable on its own.

Right. We do have skills as part of this container format, which was just called plugins.

Alessio: Mm-hmm.

Felix: And plugins are available both for Claude Code and Claude Code work the same format, and you can install plugins. This works in cowork today. You can basically say, I’m gonna add a whole, like just a GitHub repo as a.

Skills marketplace or like a plugin marketplace. And that’s how we’re doing portability. I think we have a lot of room left to grow in. How do we make it easy for people to know that they can write skills? How do we make it easy for them to just like, share a skill with you? Because obviously all the words I just said, right?

Like I’m losing most of the knowledge worker base out there, right. And start by saying, oh, you can connect to GitHub repo. It’s not exactly how most people will end up working in like a general knowledge worker space. Um, but I think there’s something there. And another thing that’s there that I think has not really been properly explored is the, the, the combination of which part of the skill is very portable and then which part of the skill is like very personal to you.

Right. And I think that’s something we haven’t really solved as an industry. Hmm.

swyx: It’s like, which, how you wanna introduce more structure to the skill or have always have like. Public skill, private skill, you know, pair. Yeah, yeah. Kind of. I think there’s

Felix: like a, like the easiest way to do this, which is we do like use string interpolation or something.

Right, right. Yeah, yeah. Insert username here, insert like phone number, insert, like known folder, locations, that kind of stuff. Um, that’s probably clunky. That’s why we haven’t built it. Um, but I do think someone is going to come up with like an interesting way to keep everything we like about skills. The portability is just a file, it’s just marked down.

It’s just text, honestly. Right. Like a text file words. The complete lack of structure, which means you don’t need any kind of tutorial to write a skill. Just like explain it to Claude the way he would explain it to me and Claude will probably get it before I work. Mm-hmm. Right? You’re just like, for booking a flight, tell Claude how to book a flight the same way we tell him somewhere.

I just started working here today. But combine that with a very like, personal thing. Um, maybe we’ll stick with a booking a flight example. I don’t actually think. AI should be booking flights. I think the tools we have is yes.

swyx: Yeah. Finally, somebody says it. It’s the default demo that everyone’s making.

Felix: I’m

swyx: like, I even against like booking demos, it is not a good showcase.

Felix: Yeah. I’m like, I just wanna book my flight myself. But, um, I think there’s a lot of things that have a personal and a non-personal component and that’s maybe why people reach for flight booking because some things are very universal. Yeah. Super flight is usually better, right? Like few people try to book the most expensive flight.

And then some things are quite personal about like what times you prefer, which seat you prefer, which airports you prefer. Combining that and like a skill format that is actually portable, compatible, easy to understand for people. I think that would be very exciting. We just haven’t figured it out yet.

Alessio: Yeah, I think the text part every, I think everybody by now has some sort of like cloud file thing. Either Dropbox, Google Drive, whatever. So it feels like in a way it should basically like sim link. My skills into all my agent harnesses. Yeah. Just keep those ing like we have internally this like valuable tokens repo, which is like all the commands sub agents.

It’s good. Uh, and then I build like a TUI where you can start it and be like, you know, install this command and this three sub agents into this agent in this folder and just copy paste this. It doesn’t do anything. It literally cp the file into that. But I feel like there should be something similar where like whenever I go into a new thing, it’s like, hey, here’s like the link to exactly the cloud folder and just bring down these skills into this.

Yeah. Like today it doesn’t quite work like that. Like if I install a new agent, I cannot, I have to like copy paste all the skills and I don’t even know where they are.

Felix: Yeah.

Alessio: That’s like the big problem. It’s like where do I find them?

Felix: Yeah.

Alessio: Um, so I’m curious like in the future like that, that almost feels like my personal productivity thing will be my skills.

Felix: Yeah.

Alessio: Is not really the product that I use. Everybody has access to the same product. But today there’s, that just looks like copy pasting ME files, I

Felix: think so many things I, I really like thinking about agents and LLMs just as like another coworker. So many attempts have made to build documentation companies that are like, oh, we’re gonna solve oil documentation problems.

Um, I myself, like spend a little bit of time working in notion, right? I’m like deeply familiar with the concept of let’s get everyone on the same page. Mm-hmm. Right? And what you’re basically saying here is you want all your agents to be on the same page about your preferences, about the skills, about the way they ought to work and like how they ought to execute.

And I’m not sure what the right thing is going to be if it’s going to be some, some company that can say, all right, we’re as an independent body, we’re not trying to like, push into any particular product. It’s our job to be like the skill authority, and we provide, I don’t know, we’re gonna be the Dropbox of skills and we can just sim link us into all the products we want to use.

I’m not sure that’s gonna be viable business, but as, as an idea, it would be cool.

Alessio: Yeah. Yeah. I think so many things are just going away as businesses. It’s like, how am I supposed to do it? I’m not even asking somebody to make a product about it. Like yeah. I wanna personally know. And there’s things like you said, it’s like you almost wanna skill and then interpolate it between personal and work.

So if I’m booking a fly for work, it’s different than I’m booking a flight personally.

Felix: Yeah.

Alessio: In some ways, yeah. But like a lot of the scaffolding is the same, you know? Cool.

Felix: I mean, as an engineer I will tell you like, you know, technic a person to technic a person. I will just be like siblings.

Alessio: Well that’s what, that’s what I do.

We call that MD and agents that MD’s just the same how sim length. And so it is like, that works, but it feels like, yeah, I don’t know. Maybe

Felix: you can always go one, you can always tell cowork problem and then cowork will solve it for you. Just make the siblings. That’s like one way to do it.

Alessio: That’s true.

That’s true. All right. Everything is called cowork.

Felix: Uh, potentially spicy. Question for both of you.

swyx: Uh, which of these industries will go away?

Alessio: Okay, so what Felix was saying before is interesting. There’s busy like. The short term pressure of like, we need to turn these tokens into valuable things, which is I should build the last mile product that harness the model.

And then there’s the question of like, long term, which ones are gonna still be valuable? And I think you’re kind of seeing this today with like, uh, you know, the coding space in a way is kind of like everybody’s moving up and up in stack because you need more than just turning tokens into code. I think search, like enterprise search is kind of saying the same thing.

Like with G Clean and like all these different companies is like, at the end of the day, if Cowork is the one doing all the work, the search itself is like such a small part that like, I don’t know if I’m really gonna pay that much money just to do search. It’s almost like everything is like a cowork vertical.

So like how much can cowork first party support?

swyx: Mm-hmm.

Alessio: And how much can it not? I think for a lot of these things, the planning thing that you were showing do Which one? The planning. The planning.

swyx: Okay. Yeah. Yeah.

Alessio: That’s one thing where like most of the value that these agents provide is like they’re better at planning for specific tasks.

Yeah. And have better tools for it.

swyx: Yeah.

Alessio: But I think the models are now moving in that direction and they have the right harnesses and they’re on your computer. So for me it’s almost like if for the end customer trusts your startup to be the provider of that task result, then I think that works. This is, uh, something that, this is a short

swyx: spike that we’re, we’re working on.

Uh, yeah.

Felix: I think, look, I’ll, I’ll, I’ll tell you this, like I don’t think I’m the best person to like actually estimate which industry is going to be hit the hardest. But I do think that at philanthropic as a group of people, we’re deeply worried about the impact. That the tools are going to have on the labor market, especially for like junior employees that, because I think, I think it’s only honest to say that when we talk about automating a lot away, a lot of the work that we personally find annoying that we maybe think’s not the best use of our time.

In a lot of industries, that kind of work would’ve been given to a junior entry level employee. Yeah. Right. And I think it’s, it’s only, it’s only right to be really worried about that and like worry what that’s going to do in particular to people like enter the shop market.

Alessio: Mm-hmm. I have a solution for that.

Which you make them, you create simulative jobs for them.

Felix: Okay.

Alessio: So this is, this is like half joke, half true. So if you think about software engineering, when you’re like a junior engineer, you work like 1, 2, 3 years. And in those three years there’s like maybe like a handful of moments where like you really learn something.

And then a bunch of other days where like you’re not really progressing.

Felix: Yeah.

Alessio: I think now we can use AI and these models to actually like shortcut these careers and almost like simulate the early years of your work and like just make them like super dense and like these learnings, it’s like, hey, we’re working on this feature, which is like a distributed system and you need to learn this thing that might take three months at a company.

And so you take three months here, it’s like we’re just simulating the whole thing. It’s actually not a real thing. And in one week we kind of speed run through the whole thing and you kind of learn your lesson from there. And we kind of repeat that in like one year. You basically get like three years worth of like projects and experience.

Yeah. I think it’s harder for like things like sales or for things like, you know, marketing because you don’t really have a way to get the feedback loop. But I think a lot of it, it sounds kind of silly, it’s like you’re making the new effect job, but it’s almost like you go to college, right? People pay to learn how to do it, and this might feel similar where it’s like, hey, we have the.

Jane Street Simulator is like, you wanna come work at Jane Street? We’ll just put you in the simulator for like three months.

Felix: Wow.

Alessio: And you’ll come out of it. It’s like, you know, I’m ready.

Felix: So there, there is an aspect here. I’m not an expert enough to like actually know what, what is going to happen to marketing or legal or finance, right?

Like, I don’t work in those jobs and I, I don’t think I should talk about them, but I am an engineer and I think I have a pretty good idea of what engineering is like. And I think one thing we’re sort of seeing is that as a company and also as, as the public, we’re like deeply worried about entry level, but we’re also seeing more senior engineers accelerate it.

If like they’re more productive. They, they actually increase the value they provide. And the thing that I’m thinking about a lot is the fact that even before all of this happened, um, I’ve always had a lot of respect for the University of Waterloo and the, the new grads that have joined my teams as from coming from the University of Waterloo always felt like.

More ready than new grads will like literally spend their entire time at the university regardless of how good, but never actually had to work inside an environment where you have to ship things that eventually will be used by users. And I’m, I’m, I’m German. I like initially went to German University and I think the, the, the like information systems programs, there tend to be very theoretical, right?

Like I often give people the example of like trying to become a doctor, but you first have to do four years of biology and as a result when you get a new grad, you sort of have to teach them what it’s like to actually build products and to work in a company and like work with other people. And like some people will have different opinion and like, how do you do all of those things?

And the University of Ulu, it seems like they just. Spend half of their time. I dunno if it’s true, but I think it’s, it’s a year, right? They spend so much time,

swyx: part of your job, uh, a cu a curriculum to do spend a year in internships.

Felix: Yeah. They just like go from company to company. They show up on your team as like a junior engineer who spend like 20 companies.

Not really, but like, it seems like a lot of my new grads have also briefly worked at Apple, Google, Tesla. Yes. And uh, there’s a common meme where they like collect all these logos, like infinity stones, but, and they always put it on LinkedIn and it is very unclear that they’re an intern. Like Yeah, yeah, exactly.

But it does actually make them so much better compared to other new grads. And I wonder if that’s a useful model maybe for the future when we also have to like, crunch down the amount of time you have as a junior employee. ‘cause the value you have as a junior employee is going to like, be impacted.

swyx: My sort of pro young people take is that they’re, you’re more, uh, you have higher neuroplasticity, you can learn more, you have less preexisting biases.

And, uh, what I is assuming is true for you, what OpenAI often says is that. Actually it’s the, the younger, like fresh grad engineers that use Codex or their coding stuff, uh, more innovatively than the, uh, experienced engineers who have a set and preferred way of doing things.

Felix: Yeah. As I talk to people, I, I someone experience.

swyx: Yeah. So maybe you’re more AI native. Yeah. And therefore you’re, you, you get cut. But like, I think the problem is you don’t need that many of them.

Felix: I mean, philanthropic is on the record as saying we do believe that the impact on the market is going to be sizable and we do not think that people overall are ready.

Right. And we do actually think we should probably talk about it as a society much more. Yeah. I’m not sure that I’m like the individual that can add like anything useful there. But I think as societies with economists and, and governments that need to wrestle those questions in a way that is probably more meaningful than me wrestling with them, we’re probably not doing good enough.

swyx: Well, we, we’ll try to educate and then I think also just releasing frequently as, as, as you guys do, or probably maybe too frequently

Felix: Yeah.

swyx: Uh, is helping people to adjust over time. Right. Rather than one big bang thing. There’s like sort of this gradual takeoff that people are living through that we

Felix: Yeah.

swyx: Waking people up. Right.

Felix: Yeah. And I, but I think a lot of us like wondering at what point do we actually have full takeoff, right? Like at what point is there, we’re all sort of expecting this like big bang moment where things will accelerate so quickly that it becomes a self-reinforcing loop.

swyx: Mm-hmm.

Felix: And at that point, it’s sort of like off to the races and there will be no more like slowly catching up.

You notice just have cloud being so good at everything.

swyx: Yeah. It’s when cowork is training models, it’s when it’s looking at tensor board and Exactly. Weight and biases and training things.

Felix: I like we can all debate like how many years it’s away, right? Like some people make a better route, like maybe it’s 10 years away, maybe it’s a year away.

Um, I’m not entirely sure where, where I come on this time, but I’m not totally sure that ultimately it matters all that much, whether or not it happens in four or five years. If we have a decent one, certainly that’s going to happen. It’s probably something we should wrestle with.

swyx: I wanted to talk, so by the way, the, the scheduled task complete, uh, the, the, there’s the clean my desktop task complete and it did it organized by file type, which, okay.

But, you know, I was trying to get it to do more sort of thematic, like read the file, understand what it’s about, group by, uh, the, the topic rather than the file type. But

Felix: I mean, you can just follow up and have it do that. Oh yeah. Here, like it did, it is proposing That’s right.

swyx: Yeah. So it’s, it’s got some like topical things, but uh, yeah, I could probably do better.

Like, yeah, so like I probably need to give it a skill to read video files so that it understands here’s how I like to,

Felix: honestly though, like, um, I see that you’re using Opus 4.6, right? Like my recommendation for people is increasingly don’t worry about it anymore. Just like tell it what you want it to do.

swyx: Yeah.

Felix: And it’s probably gonna figure out a way to do it. It might not be the way that you like necessarily or the way that you’ve gone about it.

swyx: Videos, deeper,

Alessio: lower outsourcing, organizing all of this. So let’s fight. Yeah. Yeah.

Felix: I’m honestly like, so curious what cloud is gonna come up with.

swyx: I’ll kick that off.

I wanted to also just talk about the, the overall, uh, you know, you talk about data analysis, you talk about like, uh, your, your personal finances. You also said, uh, which by the way for us is very timely tax season, right? Like Yeah. Use cloud core for tax season. It is not responsible for any mistakes, but might as well, right?

Like it’s, it’s free knowledge work for you. Yeah. Uh, so I just like, I think cloud for finance is a big deal. Um, and this is definitely like in that mix. I wonder, is it like, do you, is it a separate team? Do you talk to them? How important is it? Right. Like, because you can also natively output Excel files now.

Felix: Yeah.

swyx: Just

Felix: talk about the

swyx: finance effort

Felix: grow. Yeah. We care about the verticals quite a bit. So we do have a dedicated verticals team. We have a dedicated enterprise team,

swyx: and those is business engineering, not sales.

Felix: It’s engineering. Yeah, yeah, yeah. It’s engineering. So we do have people who sort of come to work every single day and they, they ask themselves, how do we make co-work extremely effective for people in those specific industries?

How do we make it easier for them to understand, how do we make it easier for them to plug into this and like sort of get the same value out of it that software engineers get? I think it’s no real surprise that software engineers ended up being sort of at the forefront of the entire AI moment because so much of it is this like Rub Goldberg machine nest where like we’re already used to automating things, right?

Like it’s part of our job. Yeah. So we care about it quite a bit. I think it also like really matches what we see. Cloud being very good and as a model, I think it provides tremendous amount of value to those customers in particular because. We can do so much with the amount of data they have. Those are like data heavy industries.

Their industries for correctness matters quite a bit.

swyx: So for us of, I’ve used it to analyze my business, I just can’t show it. So

Felix: it’s two sense. I had a similar question about, about taxes. Like, I did tweet, I did tweet about the fact, I did tweet about, oh, COVID is doing my taxes. This is honestly incredible.

And, um, it’s like annoying. He is like, this is so cool, but I’m not gonna, Twitter is maybe not the audience that needs to like see my tax return.

swyx: Yeah. That way. Here, here it is. It’s it’s reading on the videos, so it’s like Yeah, it’s getting more, yeah.

Felix: How did it actually do it? I’m actually curious.

swyx: Oh, usually it just like, takes a screenshot and then it reads the screenshot vi by vision.

So this is what I do for my, my Zoom upload thing, right? Because I, I have paper club sessions that I need to upload to Zoom and I want it to automatically. Uh, title them and do show notes and everything. So it just take screenshots and try to try its best. Yeah. It wouldn’t probably benefit from transcribing, which it’s doing by, it’s operating by Pure Vision now, but it’s good enough.

Felix: Yeah.

swyx: And then I, uh, I do have to call, uh, out to Nano Banana to do images. So unless you guys do images for me, uh, I have to call other people your images.

Felix: We’re aware. We’re aware. It’s, it’s just like so fun for me because like, this is the thing that I’m increasingly doing, like increasingly curious about cloud’s, creativity and like figuring out what is great Claude’s approach is like some problem.

swyx: Yeah. Vision for everything is, is like the, the superpower, right? Like, you know, and computer use, you guys were the first to do computer use, right. And when it was launched, I was very unimpressed. I was like, it’s slow, it’s unreliable, it’s wild. How much better? ‘cause it is one year ago.

Felix: Yeah, I know. Like it was barely usable.

Yeah. I, I remember it was very usable, but is it wild how much better things have gotten? Yeah.

swyx: Yeah.

Felix: Over that one year

swyx: we went to the anthropic office because you, uh, for the launch event for computer use. Like there was like this hackathon. Yeah. And like nobody hack on computer use.

Felix: But I did see, I, I I don’t know if you’re okay with me saying that, but I did see briefly that you do have like a, like an automate Mac, SMCB server installed.

Right. Uhhuh, you use that ever.

swyx: What? Sorry? Which one? Where?

Felix: Um, if you go to your settings.

swyx: Oh, settings. Okay. Uh, where, sorry, this one?

Felix: Yeah.

swyx: Yeah.

Felix: Um, I noticed that in your connectors,

swyx: Uhhuh. Uh, I probably said it at one time, but I don’t use it actively.

Felix: Oh, okay. The

swyx: a max automated. Yeah. Yeah. So, so I, yeah, this one I really wanted to like, just automate everything in my thing.

I didn’t find, I didn’t find it super reliable.

Felix: Okay.

swyx: Why?

Felix: No, no, no question at all.

swyx: Cloud is much better writing Apple Script and executing its own Apple Script than relying on these, uh, third party tools.

Felix: Yeah.

swyx: Uh, so I’ve increased, I, I initially installed Im CP and like all these other fcps that people built, and, but now I don’t use any of them anymore.

Like just, just let cloud write its own thing.

Felix: Yeah. It’s

swyx: gonna be more custom made. We keep going up the stack,

Felix: but if using computer uses like a fairly interesting area to me, and it’s like also interesting in the sense that I don’t think we’re far away from, I don’t think we’re far away from clapping, very effective, but like using your computer and not just it’s theoretical computer.

Alessio: Mm-hmm. What’s the relationship between the user and the computer? Like, uh, there, there were some tweets about how huge some of the VMs, the Claude Cowork creates ours, like 12, 15 gigabytes and people complain. Yeah. But at some point it’s like, if you’re using the computer, you’re taking action on, it’s, it’s just your computer.

And I’m just looking at it, you know, it’s like, I, I think that’s why people like the idea of like the Mac mini and the open claw or whatever on it because it’s like, it got its own home. You know? It is doing its thing, I’m doing my thing. I think there’s some kind of like, not like risk condition, but it’s like, okay, if I kickstart this task now I can’t really use the computer.

Felix: Yeah.

Alessio: You know, because car coworkers doing things on it and it’s kind of awkward, like, yeah. I’m not sure.

Felix: I, I do think it’s a super interesting area because I, I can maybe tell you like some of the things I thought about that I think are actually a bad idea. So when, when we initially started working on cowork, I, I did have some dreams about, well, would it look like for cloud of its own cursor?

Could be cool, right? Like it’s a computer, we can write code, we can touch everything. Like who says that computers need to have one cursor? We could do a second cursor, but that actually breaks down quite a bit. Even if you go and like present cool dreams to both Apple and Microsoft, you’re like, wouldn’t it be cool if, um, it breaks down quite a bit?

‘cause so many of our models on a computer are built around this idea of like, there’s only one thing working on it. Yeah, there’s like a foreground app, a background app, cloud and Chrome can work in the background, but that’s like within one application. But the operating system layer, that is a lot harder to implement.

So I’m, I’m still grappling with what, what does it mean for cloud to actually act on your computer. It’s the right format for cloud to have its own computer that you set up. And maybe every now and then you like zoom in and you play with it. Or is the right format for Claude to just like, wait until you are.

Stepping away for a little bit and take over while you’re gone. Or it’s the right move for cloud. Just like if it’s on computer in the cloud, and like whatever you want cloud to do, you have to set up yourself. Right. There’s like a, there’s like a number of different options. Um, this is the thing I think about a lot, like what is the relationship between you and your computer and you and your data on their computer?

Because how intimate that relationship is kind of depends on the tool and Right. The thing that you’re current looking at, right? Like we’re quite comfortable sharing some things, very uncomfortable, sharing other things. And I think whatever product is gonna be successful, we’ll have to deal with those, like, with those different things.

But you probably, even if Claude was capable of making a determination, would you want Claude to make that determination in the first place? It’s tricky, Barry, because it’s like, it’s more than just privacy. It’s like almost intimacy and it’s like tricky to reason about in a way that will make everyone comfortable.

Alessio: Yeah, I could see. You know, a virtual box, like actual virtual box app where like you run the VM and then you have like a screen within the screen, you know, you can put it in the background, but then you can like jump in the screen and like you,

Felix: that’s not a bad idea. Yeah.

Alessio: You know, like, I mean I used it, you know, people used to do it virtualizing like C Linux in a Windows machine.

Felix: Yeah.

Alessio: And like you would just jump in and then you would jump out. But it’s like, it’s not like a dual boot. It’s like within the thing. The problem is that you need twice the amount of ram, twice the amount of, you know, it’s like, it’s kind of taxing on the machine. But I think that would be cool. Kinda like see, you know, the little quad window.

I can see desktop look cute. It is clicking around things

swyx: I was gonna bring up. He’s the original machine and the machine guy, because he has the uh, windows. Windows 95 project. Where’s, where’s the Windows 85 project at?

Felix: It’s probably somewhere in my GI guitar,

swyx: right? No, no, no, no, no. It is like the first thing you see is this one.

Nice. Yeah,

Felix: yeah,

swyx: exactly.

Felix: That was honestly a very fun project though. Like, obviously I didn’t, I, I should say this, just so that No, it’s the wrong impression. I did not write the actual, the actual, obviously I didn’t build Windows only five because I was a child, but also I did not build the actual engine that is capable of like simulating an X 86 processor and JavaScript and m um, that’s a tool called V 86, which is very cool and everyone should try.

But this came out of a, this came out of like a debate we had at work where people were like, they often are in the into debating the merits of electron and whether or not we should be building software in JavaScript, yes or no. And I still am very upset that I can run all of Windows 95 in JavaScript.

And launch Microsoft Excel inside the virtualized JavaScript Windows only five machine, and do things that pro, I can do that entire chain faster than I can do a lot of other things in like traditional SaaS applications. Mm-hmm. Uh, this is sort of like a, like a performance rampage that I went on. So I’m mostly built this as a joke for some of my colleagues at Slack.

This took, took like one night. Um, what, but then that I, it was, it was not hard to do. It was all the hard work is in V 86. Yeah. Like if, go to the repo, it’s gonna say like, 99% of his work is done by, by um, a guy who goes after the, by the name. Copy. His name is Fabian.

swyx: Yeah.

Felix: Um,

swyx: cool. I think you’re, you’re kind of back on the Windows grind ‘cause you’re building out the Windows support.

Uh, I thought there was some really cool technical stories to tell. Uh, and it gives people an appreciation of like, well here’s how hard it is and here’s how important here, how, how you invested the sandbox. So maybe this is like a good opportunity to talk about something in the details.

Felix: Oh yeah, the, the VM honestly is like so cool.

There’s a lot of things we dislike about the vm, right? Like there, there’s a lot of things that are real trade offs and you want to know why you making those trade offs. Um, and you’re right, like a lot of people write me like, Hey, how, how come cloud is taking up 10 gigabytes? I could say on the point, it’s not actually taking up 10 gigabytes.

It’s just like a way that macros displays bites is like wrong, but the way we actually ride it to disc is by we collapse the empty space and the image, so it’s not actually taking up 10 gigs. But that’s a technical differentiation. That’s probably not gonna matter to, like,

swyx: to me, the the, the outcome is it takes too long to start.

Yeah. It’s like 30 seconds sometimes. So I don’t know. Oh, it should be faster than that. Whatever it be te about this feels like 30.

Felix: Yeah. Like even either way, like whatever it is, it’s going to be, it’s going to be slower than just running Log Ultra on your computer. Right. So the trade offs are real, but what we’re doing on Windows, we’re using the Windows, windows, uh, host compute system.

It’s the same thing that WSL two runs on, like the Windows subsystem for Linux that I think a lot of developers appreciate quite a bit. Yeah. Um, and it’s, it’s pretty cool because we sort of like have to separate out which system space the virtual machine runs in, in who gets to talk the virtual machine because obviously you give this virtual machine a decent amount of power.

How do we optimize not just the connection between the two systems, but also how do we make sure that random other application doesn’t get to talk to Clot inside the vm?

swyx: Hmm.

Felix: We do some pretty interesting things. Um, last week we started writing a new networking service. A networking driver. That optimizes how Claw talks to the internet.

If your company’s doing like weird internet things like pack inspection and like, like, you know, taking your part as a cell and inside your company, I think there was probably like a very small, easy version to build of cowork that is much simpler but also breaks on most com most users, computers. And this one is quite nice because it works on most users computers.

Um, and the default example I always go for is I, I really want this to be highly effective on like a, on like a machine that most people pick up. And that machine will probably not have Python, it will not have no j And even if I just take away those two things, cloud is going to be so much less effective from

swyx: your computer.

So what do you do? You don’t even, I mean, may maybe require people to install Node in Python.

Felix: Oh, like, you mean for like a, what does the feature look like without a vm?

swyx: No, no, no. So, so like, like you said, right? Let’s say a target machine is whatever’s a default spec, windows laptop.

Felix: We do this, which is quite cool.

So on, on, uh, mes, we use the, um, apple virtualization framework, which is pretty solid, optimized, like it’s good stuff, and instead simple a p call, right?

swyx: It’s

Felix: like super simple.

swyx: I, I saw the code recently and I’m like, that’s it. What the f**k

Felix: would you, once you start like shipping production code on it, you start adding like all of these edge cases, your new

swyx: Oh

Felix: yeah, it ends up being a little longer, but, um, I think Apple really cooked with a virtualization framework and it’s very, very good.

It is very fast, it’s very reliable. And same on Windows. The, the host compute system. I think WSL two as well is maybe one of the diamonds within Windows. It’s like one of the few things that developers universally rave about is very, very cool. And like hooking into the same subsystem makes a lot easier for us to say We don’t really care how locked down your computer is.

Maybe it’s like your employer’s computer and your employer has decided that you get to install nothing.

Alessio: Mm-hmm.

Felix: Not trusted, but it’s true in a lot of environments, right? Like even at Anthropic, um, our IT department controls what kinda stuff you install, just like a pretty common experience for many companies.

Um, and this gives it departments a decent amount of, like, it makes their job so much easier because we can say you can separate out cloud’s computer from the user’s computer. And then for cloud’s computer, where you probably care about its data loss, you care about like a potentially hostile actor, you care about maybe data being exfiltrated.

And once you control the network and the file system layer, you don’t really care necessarily anymore. That cloud might be writing super useful Python scripts. What worries you about the fact is that like once you install Python, now anyone can do anything on a computer. Once you put that in the vm, that risk really goes down.

swyx: Yeah.

Felix: So that’s why we jumped through all of these hoops.

swyx: Yeah. I think you, you had a different, uh, tweet about this. Um, but it, it’s, it’s almost like people have also approved exhaustion. Like, it’s like you can’t approve every single commands. Like sometimes by, by default, some of the theis, I think even early called code, uh, we have to approve every single command.

Yeah. And, and like it’s so, so there’s this sort of dichotomy between either approve every step or dangerously get permissions.

Felix: Yeah.

swyx: And actually sandboxing is like, kind of like the middle ground.

Felix: Yeah. I do think, I do think it, it’s maybe on us as like the AI industry to come up something better than, oh, this is super safe as long as it doesn’t do anything right.

Right. But if you want this to be useful, then you have to like approve every single step of the way. And like, computer use is a good example. The only way to make computer use on your host, like super safe, like really super safe is probably if you approve every single action, right. Like models, like, I would like to type the word.

You’re like, okay, that seems fine. I know, I know. Which, like cursor is focused. Yeah. It’s not

swyx: automation if you don’t delegate.

Felix: Yeah, exactly. You need to like properly delegate. You need to be able to like delegate and walk away and trust that this thing is not gonna like mess dramatically. And I don’t even think we need to build perfect systems.

I don’t think we need to wait for like a hundred percent model alignment. We can rely on the same Swiss cheese model we’ve used in the industry for a long time. But I do think we need to like universally maybe eventually invest more. And that’s what we’re doing. We need to invest more in systems where we can say, you do not need to approve everything.

swyx: Speaking of Swiss cheese model, he just wrote a thing about this.

Felix: Oh cool.

swyx: Yeah. Uh, yeah. Um, yeah. Super cool. I mean, yeah, it’s, it’s weird how like, I guess usually I think safety and security is kind of like a boring word to, to engineers. They’re like, just gimme be unsafe, gimme unsecure. But, um, I think.

Achieving the right thing. Like you are going after a consumer slash prosumer.

Felix: Yeah. Yeah. Talking both kind of like both. I think I, I also want to capture people who would’ve no trouble using clock code like yourself, right?

swyx: Yeah. Yeah.

Felix: But still find it maybe just convenient, easier. You’re like, oh cool.

That’s like the list on the right. I can edit it. Those things are just easier to do if you have

swyx: to. But this is like clearly the knowledge work side. Yeah. Claude Code will clearly capture the development workflow. But like I, I, I do think like you have to sweat this like safety and security details in order for people to trust it.

And like the even Claude and Chrome, like having the whatever API uses to do the background thing.

Felix: Yeah.

swyx: Um, that’s the only reason I use it is because otherwise I would have to just get a separate machine.

Felix: Yeah.

swyx: And just run it, run to the, and that sounds like

Felix: super annoying.

swyx: Yeah. I mean, like currently doing it, but,

Felix: and I think, I think also as developers, um, maybe we’re, we are more risk tolerant, but we’re also just like accepting we are more risk tolerant, but I think we also just have.

I don’t wanna say arrogance, but like sort of the trust that if like the really bad thing happens, we can probably fix it.

swyx: I just tell Claude to like, check with me before doing any irreversible action. Like sending an email or doing permanently. Yeah, it’s good enough.

Felix: But like, not even Claude, I mean like simple things such as NPM install, right?

Like we’re all running NPM install with full user permissions and if it wants to like read SSH, it well crazy that that is the default kind of why. Yeah, I know. I agree. I agree. Fine. Like I’m obviously doing it every single day. No, right. Like, uh, and I think obviously NPM and GitHub too have like done a pretty good job maybe over the last couple months to like clean house and come up with like more specific tokens.

But generally speaking, I think as engineers we’ve always been a little bit more risk tolerant. And if you do a little bit of introspection and you ask yourself, is that how we should be doing things, you might not always come up with the right answer. And I think for models too, like my approach, like I’m not gonna, the the safest thing is to do nothing.

We do want products that are quite capable, but to the extent possible, I don’t wanna ask you, are you okay with the script? Because I kind of believe that once it starts becoming a part of your workflow, you’re probably not either, either you don’t have the skill to understand whether or not the python, the script is safe or you’re not gonna read it anyway.

swyx: Cool. I guess a, a couple partying questions. Uh, what’s the future of clockwork?

Felix: I think we’re still, we’re still such early days. We’re gonna keep shipping things that we’re gonna keep shipping, things that, um, we’re gonna keep iterating on this thing like pretty quickly, but, which I mean, you can sort of continue to expect that every single week there’s gonna be like a small new feature, if not a big new feature.

Um, I’m going to continue probably to double down on your computer and like making you effective in your computer and making cloud effective in your computer. Um, we’re starting to grapple, as we talked about today, grapple more with a question of like, what does it mean? What does your computer mean? Does it have to be the one in front of you or like a VM on your computer or like a computer somewhere else?

And then the third thing that I’m quite excited about is. We’re continuing to go off this hill climbing on slowly taking users who are used to asking questions and getting an answer to slowly teaching them to like step more and more away. And that claw take over like bigger and bigger tasks and work both in time as well as in like scope.

And I think you can probably see most of the, our investments on our feature releases to like work on both of those things, like the ability to do more on your computer and then the ability to do more independently for longer.

swyx: Does remote control work for Claude Cowork yet? No. Right.

Felix: Excellent question.

swyx: Coming soon. I mean, that’s an obvious thing if you want to keep betting on the, on your computer, but I, to me like. You know, we, we talk about like, people are not ready this year. Like the, there’s, there’s no wall. It’s, it’s accelerating to me like what will be we be doing differently at the end of this year that, you know, we are maybe not even thinking about this, uh, at the start of this year.

Right. Like, I’m just trying to look ahead as to like, what, what’s like a good use case that you’re, that we sort of aim towards? So for, for example, for the machine learning scientists, it’s always, okay, well I want AI scientists, I can automate, automate machine learning, but like for, for knowledge work, I mean, I can already, you know, get it to sign up for Google Cloud to mean as a GI.

Felix: Yeah. ‘

swyx: cause Google cuts are, but like, what, what is, what’s beyond that? I don’t know.

Felix: I think it’s basically the idea that like you still had to tell her to build your script, right? He was still kind of involved.

swyx: Yes.

Felix: In maybe a way that felt kind of magical to you, but like, maybe to me on the other side is the person building this product still feels kind of heavy handed.

I see so much process that I’m like, oh, lemme take that away from you. Okay. But like, how do I just go, I will continues to go or continue to go like further and further up the stack. Make your life easier and easier.

swyx: Oh, here’s one. Right?

Felix: Yeah.

swyx: Watch, uh, I, you know, I don’t care about my own privacy or whatever, or I trust cloud, I trust philanthropic.

So just watch everything I do on a normal day-to-day basis. At the end of the day, tell me what you is called co workable.

Felix: Yeah. I

swyx: dunno.

Felix: I think the funny thing about a lot of these products is that like, for good reason, I don’t enjoy, I, I don’t, throughout my entire career, I’ve never like teased too much what I’m working on because I think you should just like, yeah.

Release it. Yeah. Build the base and release it, and then talk about it. Like I’m, I’m not a big fan of the like vague posting my own work ahead of time.

swyx: Yeah.

Felix: But the thing that is like always so fascinating to me is like, both of you all multiple times a day, you’ve like mentioned things and I’m like, yeah, that is obviously like very obvious

swyx: Okay.

Felix: That someone should be working on those things. Um, and I think we’re still in the space where if you look at cowork. The things that we will be releasing will probably not be a big surprise to either of you. You’re gonna be like, yeah, obviously that’s valuable obviously that we’re working on those things.

swyx: Yeah.

Yeah.

Felix: And obviously that’s good and useful. And the more I hit those points, the more our features fit into that category, I think the better it is for us because then we don’t end up building things that are too hyper specialized to difficult harness style.

swyx: Yeah. I think the hyper specialized thing is very important.

It keeps you like general purpose. It, it means you’re not thinking too small. Maybe I don’t, I don’t know what the, the word is.

Felix: Yeah, yeah, exactly. It’s like the whole concept that like at no point if we release, you know, there’s no Claude Code for no jazz applications that use React and 10 Stack. I know any of those two things.

And like if it’s anything else, I know several startups like that. I think that’s pretty, like, I’m not a vc, I’m not an investor. It’s like hard for me to predict where the markets go. But in terms of the building box that I’m interested in, the electron is probably by far the most popular thing I ever built.

And, um, electron itself is like. Very abstractable and generalizable. Right? Like so many apps run in it. And I think it would’ve been hard for me to predict how many apps actually end up using Electron.

swyx: Yeah.

Felix: Um, and what would’ve been even less useful for me to predict this in what those apps do. I distinctly remember a bloom coming out of being like, that is cool.

Like you are a camera in a little circle in the corner. That is pretty smart.

swyx: That’s an app. Yeah. Yeah.

Felix: Or at least was, I’m not sure if it still is. It was for a while. Or like one password has so many interesting things. Right. It, it’s, it’s, it’s a level of the stack that I’m quite comfortable with. And whenever I give other engineers, advisors actually that layer that I think is most valuable to invest in because the tools of that layer are not that good.

But that’s where you get the most leverage

swyx: for like,

Felix: the future in general.

swyx: Just quick tangent on Electron. ‘cause I always wonder this, uh, have you looked at Tori?

Felix: I have, yeah.

swyx: What’s your take? Uh, you know, look, my, my my, my view is like most things should be Tori by default, unless you really need the full power of electron, but.

Felix: Yeah, I can give like my take on, I can give my big take. Why do we ship an entire version of chromium inside the thing, right? Like why do we do that? And, um, people ask me this question a lot because it’s like very counterintuitive. Wouldn’t it be much easier to use the web use that are on the operating system?

Wouldn’t it be much easier not to have to do that? And the answer is yes. And like obviously I did that once upon a time. I did that there was a version of the Slack app that used just the operating system that use Wait, did you, did you start the Slack app? I would, well, team effort and

swyx: Yeah, but I was, I was there.

We built the Slack app.

Felix: Yeah. It’s crazy. Um, I mean obviously you get the electron guy to do it, but, well, but this is an interesting point. Like, by the time, by the time I joined Slack, they already had an app that was built with something at the time called Met Gap. It was a little bit like the same app gap thing for mobile.

It just used the operating systems. Web views. Um, and that didn’t work for like so many reasons. Um, and they were like, all right, maybe we need like bigger guns. We need to like take more control of the rendering stack. And there’s, there’s a few things I always mention here. Um, I think if you’re building a small app, just going with the operating systems web view is perfectly fine.

If you’re building an app, maybe that doesn’t have too many users who will like cry bloody murder. If it doesn’t work, that is fine. The reason to go with your own embedded rendering engine is because, and this is still true in 2026, the operating system render engines are not that good. They’re just not that good.

Both Microsoft and Apple are trying to move away from that. They so far really haven’t, the only way to upgrade those is to upgrade your operating system. So if you are, say Slack and you have critical rendering bug in WK WebU and some of the other WebU options, your only recourse is to tell your customer, oh, sorry, you’re too poor.

You didn’t bother the, its MacBook. Unacceptable.

swyx: Mm-hmm.

Felix: Unacceptable to user, unacceptable to user developer. So you sort of need to like go down the stack and like find the best rendering engine, then put it in your app. Why chromium, even though it’s very big chromium is by far the best thing. Like I, I often like to remind people the unreal engine, you wanna render some text.

They use chromium. Like chromium is part of the unreal engine for same purposes. Chromium is very, very good. I think it’s like one of the marvels of engineering. It’s very hard for, we’re in San Francisco right now where we’re recording. Most of the people in the city are web developers. It’s hard for me to like overstate how magical it is.

They run seat like rendering a YouTube video dynamically. Negotiating a bit rate, figuring out what to do about your extremely broken hardware driver. Actually, this is a fun thing. Um, okay, you can enter Chrome call on Wack Wack GPU. Okay? And if you scroll down a little bit, these are all the enabled workarounds because something is going wrong on your computer.

If you’re doing this on a Windows computer with like A GPU, that is not the most popular GPO, it will be much longer. And all of these are usually just there to make sure that if I say as a developer, I want a red pixel to appear here, that that actually happens. Chrome is such a marvel because of works on all the machines that user might throw you and it’s gonna work fairly reliably.

And if it doesn’t, they will probably fix it within 24 hours.

swyx: I see. So this is the super operating system, right? That that works everywhere.

Felix: Yeah.

swyx: Right. Okay. Yeah.

Felix: So a lot of the magic of Electron is honestly just that it makes it very easy for you to ch chromium in a way that serves you exactly in your use cases.

Elect, uh, exactly.

swyx: Our next interview is with Morgan Dreesen.

Felix: Yeah.

swyx: Who had the phrase like, desktop OSS are just poorly deep, uh, poor implications of the, the actual os, which is Chrome, which like actually works everywhere. And this is this, this is the platform where you ship apps.

Felix: I, I think the wild thing is that like as engineers, we so often sort of assume that the platform, like the layer below us is like super stable.

Mm-hmm. And then you talk to those people and they’re like, ah, we are also just like guessing. Um, uh, and I had like a distinct moment at Slack where one of our customers at Slack was Nvidia, and for a while I really put GPU developers on this pedestal in my head. And I do think they’re still probably much smarter than I am.

But I was like hardware engineers who built the chips, who then like built the drivers. Their work must be so much harder than mine. They must be very good. And we had like one bug in Slack where like if you had a YouTube video in Slack, it wouldn’t quite render why. Like it would have these weird artifacts.

And, um, that ended up being a chromium bug. And I ended up on this like giant thread. So I got to see a lot of the source code. And they also are just like common to do. We don’t know why this is weird, but if you flip this bit, things work. You know, this is just like happening with every layer of the stack.

Maybe the, uh, you know, the,

swyx: the end of year a GI prediction is that clock can build chromium. You see, you see you, you laugh now. But yeah, like, you know, someday

Felix: it’s, it’s sounding, it could get pretty good. Like it used to be completely useless. Um, mostly just like overwhelmed, both with how hyper specialized tools are inside the chromium repo.

Like for, for a long time. Chrome has like sort of reinvent all the tools because none of them are capable of ending Chrome. I think the EGI moment I am kind of waiting for is at what point are we gonna say Electron is probably no longer necessary because you can just build fully native apps. The Swifty?

Yeah. Like not just in Swift because this is one thing, like it’s pretty easy if you, I think our current models are quite capable of taking an electron app and replicating it Swift, are they gonna be capable of like building an app that is actually more performant, which is less memory? All of that stuff, um, is gonna go into the same hyper optimization that developers have done for like a long time.

We’re not quite there yet. Work and like point even our best models at a thing and say, just replicate this, a native code. Make no mistakes. Ultra think. Right? We’re not quite there yet. Um, ultra

swyx: think is bad

Felix: today. Think is back. Yes. Okay.

swyx: Or we’ll get an ultra think for like days,

Felix: just a pretty long time before,

swyx: but he worked on Ultra think for days.

Yeah. Why he just, it’s just. Front,

Alessio: I’ll let it, the

Felix: more goes into

swyx: it. Yeah. Okay.

Alessio: Another question I had is like coworks. So if I have my Claude Cowork, like what’s kinda like the multiplayer mode? I think sub agents is like single player Split up the context.

Felix: Yeah.

Alessio: And the multiplayer cowork is like, my colleague is some file on their machine that I wanna know about or I wanna know how their task is going to then update my thing.

Like is that interesting? Is that something that makes sense for you to build or for like

Felix: It’s like super interesting to me it, it almost goes back to like some of the scaffolding room. Like okay, are we gonna be end up, are we, will we end up building scaffolding that will just go away? And like a question I have here is at what point do we just assign these things, like their own Gmail account?

We just give them their like Slack handle and then they will just like use the same tools we humans use to interact with each other. You mentioned our finance people, they’ve been working pretty hard on very good office integrations. And I think for a while we’ve been like, we built so much tech around cloud, leaving useful comments inside a Google Doc, and now it just does, it just like leaves a comment in your Google Doc and that’s how you interact with it.

Maybe like the similar thing where I still have open questions around what is the best interaction mode? Is it for us to build something super custom for cowork agents to talk to each other? Or is it okay, let’s just jump straight to the finish line and say, well, we’re just gonna give this thing, if you use Slack at work, we’re just gonna give this thing a Slack handle.

And that’s going to be the way, it’s like multiplayer capable.

Alessio: They communicate with each other. Yeah. Yeah. Like, you know, as a, as a fun project, I build this thing called piq, which basically takes any repo and the PI agent, uh, coding agent, it puts it in a VPS, and then there’s a public web hook where anybody can submit a coding task.

Oh. And then there’s a dashboard in which you review the task and then piq pi, pi, uh, queue.

Yeah. You basically get all these like tasks, anybody can submit a task.

Felix: Mm-hmm.

Alessio: And to me it’s almost like in the organization of the future, it’s like the sales people are talking to the engineering team that is talking to the marketing team, to the product team, and all these coworker are going to like queue up decisions for other people to approve in a way.

Felix: Yeah.

Alessio: You know, and I’m kind of curious what that looks like and like how do you, how do I give my cowork the ability to build a proof task without asking me

Felix: Yeah.

Alessio: And how to decide which one I need to review. Yeah. You know, because for some of these things it’s like, you know, you wanna change the color of something that’s kinda like a branding decision.

Or another one is like, hey, your thing is just broken. It’s like, this is like how you fix it. Yeah. And Claude can actually review whether or not that prompt matches what he’s trying to do today. Everything is still very, it’s like multiplayer within the single player, you know? Yeah. I guess spin up many of them, but like, how do I get multiple people to hand off to each other things using their particular context?

Felix: Yeah. And for both of your coworkers to like talk to each other. Right,

Alessio: right. Yeah. Hey, we got an episode today. Can you like, have you, you know, or

Felix: Yeah. This is like a, uh, I know we’re like running out of time here, but like we, we previously talked about sharing skills and I did have this question of like, what if your cowork would just like ask the other coworks if they have a skill for this task?

Doesn’t matter. These could do.

swyx: Right. Like, okay, so skill transfer.

Felix: Yeah, like,

swyx: um, and again, that’s, maybe

Felix: this maybe goes back into the territory of like building something very powerful and building something creepy often goes hand in hand. Um, because I could tell from the reaction that my fellow engineers said that this is probably not what we’re gonna do, but like.

We have Bluetooth le right? Like I, this computer can figure out that it’s sitting right next to this computer. So you’re probably working on the same thing. Um, well, you see that in cowork, probably not. But, um, there’s like, I think really creative solutions to problems that we really haven’t tried yet.

Yeah,

Alessio: yeah, yeah. Yeah.

swyx: Excellent. I guess the, the last thing is, uh, philanthropic labs. Uh, I always have this mental model of a model lab versus, uh, agent lab. And this is basically Anthropics internal agent lab, which co Claude Code, uh, is now under, right? It’s part of the whole org.

Felix: I mean, people are so fungible, right?

Like,

swyx: okay, this is just, I, I don’t know how, I don’t know real. This is, I don’t know.

Felix: No, it’s a real team. It’s a very, um, the, the last team is primarily working though on things that you don’t see in public yet. Um, they’re trying like really wild out there, ideas that seem quite improbable. Um, the mad science

swyx: thing.

But you, you’re, are you officially under this thing or

Felix: No? We’re, where is the Claude Code is, but now Claude Code is like a fairly big group where. I actually know many people we are like, like I remember yesterday coming into our weekly COVID meeting. I was like, woo,

Alessio: this is hot.

Felix: There’s a lot of people here.

Um, but we still have a labs team and we actually made the labs team a lot bigger. Mike just joined the labs team as a, as an ic, which I think is very cool and very fun. But they’re, they’re working on things that you have not seen yet that are extremely out there and probably half broken. Right? Like the sort of the idea of a lab team is that it should only work on things that make really no sense for anyone else to work on.

swyx: Okay. Well, looking for exciting things from there, but thank you so much. I know we’re out of time, but uh, appreciate your joining us. I appreciate co cowork, everyone go use it. Uh, it is the closest I’ve felt to a I this year. That’s so nice you to say. Thank you very much. Yeah. Thank you for your time. Yeah.

Turbopuffer came out of a reading app.

In 2022, Simon was helping his friends at Readwise scale their infra for a highly requested feature: article recommendations and semantic search. Readwise was paying ~$5k/month for their relational database and vector search would cost ~$20k/month making the feature too expensive to ship. In 2023 after mulling over the problem from Readwise, Simon decided he wanted to “build a search engine” which became Turbopuffer.

We discuss:• Simon’s path: Denmark → Shopify infra for nearly a decade → “angel engineering” across startups like Readwise, Replicate, and Causal → turbopuffer almost accidentally becoming a company • The Readwise origin story: building an early recommendation engine right after the ChatGPT moment, seeing it work, then realizing it would cost ~$30k/month for a company spending ~$5k/month total on infra and getting obsessed with fixing that cost structure • Why turbopuffer is “a search engine for unstructured data”: Simon’s belief that models can learn to reason, but can’t compress the world’s knowledge into a few terabytes of weights, so they need to connect to systems that hold truth in full fidelity • The three ingredients for building a great database company: a new workload, a new storage architecture, and the ability to eventually support every query plan customers will want on their data • The architecture bet behind turbopuffer: going all in on object storage and NVMe, avoiding a traditional consensus layer, and building around the cloud primitives that only became possible in the last few years

• Why Simon hated operating Elasticsearch at Shopify: years of painful on-call experience shaped his obsession with simplicity, performance, and eliminating state spread across multiple systems • The Cursor story: launching turbopuffer as a scrappy side project, getting an email from Cursor the next day, flying out after a 4am call, and helping cut Cursor’s costs by 95% while fixing their per-user economics

• The Notion story: buying dark fiber, tuning TCP windows, and eating cross-cloud costs because Simon refused to compromise on architecture just to close a deal faster

• Why AI changes the build-vs-buy equation: it’s less about whether a company can build search infra internally, and more about whether they have time especially if an external team can feel like an extension of their own • Why RAG isn’t dead: coding companies still rely heavily on search, and Simon sees hybrid retrieval semantic, text, regex, SQL-style patterns becoming more important, not less • How agentic workloads are changing search: the old pattern was one retrieval call up front; the new pattern is one agent firing many parallel queries at once, turning search into a highly concurrent tool call • Why turbopuffer is reducing query pricing: agentic systems are dramatically increasing query volume, and Simon expects retrieval infra to adapt to huge bursts of concurrent search rather than a small number of carefully chosen calls • The philosophy of “playing with open cards”: Simon’s habit of being radically honest with investors, including telling Lachy Groom he’d return the money if turbopuffer didn’t hit PMF by year-end • The “P99 engineer”: Simon’s framework for building a talent-dense company, rejecting by default unless someone on the team feels strongly enough to fight for the candidate

—Simon Hørup Eskildsen• LinkedIn: https://www.linkedin.com/in/sirupsen• X: https://x.com/Sirupsen• https://sirupsen.com/aboutturbopuffer• https://turbopuffer.com/

Full Video Pod

Timestamps

00:00:00 The PMF promise to Lachy Groom00:00:25 Intro and Simon's background00:02:19 What turbopuffer actually is00:06:26 Shopify, Elasticsearch, and the pain behind the company00:10:07 The Readwise experiment that sparked turbopuffer00:12:00 The insight Simon couldn’t stop thinking about00:17:00 S3 consistency, NVMe, and the architecture bet00:20:12 The Notion story: latency, dark fiber, and conviction00:25:03 Build vs. buy in the age of AI00:26:00 The Cursor story: early launch to breakout customer00:29:00 Why code search still matters00:32:00 Search in the age of agents00:34:22 Pricing turbopuffer in the AI era00:38:17 Why Simon chose Lachy Groom00:41:28 Becoming a founder on purpose00:44:00 The “P99 engineer” philosophy00:49:30 Bending software to your will00:51:13 The future of turbopuffer00:57:05 Simon’s tea obsession00:59:03 Tea kits, X Live, and P99 Live

Transcript

Simon Hørup Eskildsen: I don’t think I’ve said this publicly before, but I just called Lockey and was like, local Lockie. Like if this doesn’t have PMF by the end of the year, like we’ll just like return all the money to you. But it’s just like, I don’t really, we, Justine and I don’t wanna work on this unless it’s really working.

So we want to give it the best shot this year and like we’re really gonna go for it. We’re gonna hire a bunch of people. We’re just gonna be honest with everyone. Like when I don’t know how to play a game, I just play with open cards. Lockey was the only person that didn’t, that didn’t freak out. He was like, I’ve never heard anyone say that before.

Alessio: Hey everyone, welcome to the Leading Space podcast. This is Celesio Pando, Colonel Laz, and I’m joined by Swix, editor of Leading Space.

swyx: Hello. Hello, uh, we’re still, uh, recording in the Ker studio for the first time. Very excited. And today we are joined by Simon Eski. Of Turbo Farer welcome.

Simon Hørup Eskildsen: Thank you so much for having me.

swyx: Turbo Farer has like really gone on a huge tear, and I, I do have to mention that like you’re one of, you’re not my newest member of the Danish AHU Mafia, where like there’s a lot of legendary programmers that have come out of it, like, uh, beyond Trotro, Rasmus, lado Berg and the V eight team and, and Google Maps team.

Uh, you’re mostly a Canadian now, but isn’t that interesting? There’s so many, so much like strong Danish presence.

Simon Hørup Eskildsen: Yeah, I was writing a post, um, not that long ago about sort of the influences. So I grew up in Denmark, right? I left, I left when, when I was 18 to go to Canada to, to work at Shopify. Um, and so I, like, I’ve, I would still say that I feel more Danish than, than Canadian.

This is also the weird accent. I can’t say th because it, this is like, I don’t, you know, my wife is also Canadian, um, and I think. I think like one of the things in, in Denmark is just like, there’s just such a ruthless pragmatism and there’s also a big focus on just aesthetics. Like, they’re like very, people really care about like where, what things look like.

Um, and like Canada has a lot of attributes, US has, has a lot of attributes, but I think there’s been lots of the great things to carry. I don’t know what’s in the water in Ahu though. Um, and I don’t know that I could be considered part of the Mafi mafia quite yet, uh, compared to the phenomenal individuals we just mentioned.

Barra OV is also, uh, Danish Canadian. Okay. Yeah. I don’t know where he lives now, but, and he’s the PHP.

swyx: Yeah. And obviously Toby German, but moved to Canada as well. Yes. Like this is like import that, uh, that, that is an interesting, um, talent move.

Alessio: I think. I would love to get from you. Definition of Turbo puffer, because I think you could be a Vector db, which is maybe a bad word now in some circles, you could be a search engine.

It’s like, let, let’s just start there and then we’ll maybe run through the history of how you got to this point.

Simon Hørup Eskildsen: For sure. Yeah. So Turbo Puffer is at this point in time, a search engine, right? We do full text search and we do vector search, and that’s really what we’re specialized in. If you’re trying to do much more than that, like then this might not be the right place yet, but Turbo Buffer is all about search.

The other way that I think about it is that we can take all of the world’s knowledge, all of the exabytes and exabytes of data that there is, and we can use those tokens to train a model, but we can’t compress all of that into a few terabytes of weights, right? Compress into a few terabytes of weights, how to reason with the world, how to make sense of the knowledge.

But we have to somehow connect it to something externally that actually holds that like in full fidelity and truth. Um, and that’s the thing that we intend to become. Right? That’s like a very holier than now kind of phrasing, right? But being the search engine for unstructured, unstructured data is the focus of turbo puffer at this point in time.

Alessio: And let’s break down. So people might say, well, didn’t Elasticsearch already do this? And then some other people might say, is this search on my data, is this like closer to rag than to like a xr, like a public search thing? Like how, how do you segment like the different types of search?

Simon Hørup Eskildsen: The way that I generally think about this is like, there’s a lot of database companies and I think if you wanna build a really big database company, sort of, you need a couple of ingredients to be in the air.

We don’t, which only happens roughly every 15 years. You need a new workload. You basically need the ambition that every single company on earth is gonna have data in your database. Multiple times you look at a company like Oracle, right? You will, like, I don’t think you can find a company on earth with a digital presence that it not, doesn’t somehow have some data in an Oracle database.

Right? And I think at this point, that’s also true for Snowflake and Databricks, right? 15 years later it’s, or even more than that, there’s not a company on earth that doesn’t, in. Or directly is consuming Snowflake or, or Databricks or any of the big analytics databases. Um, and I think we’re in that kind of moment now, right?

I don’t think you’re gonna find a company over the next few years that doesn’t directly or indirectly, um, have all their data available for, for search and connect it to ai. So you need that new workload, like you need something to be happening where there’s a new workload that causes that to happen, and that new workload is connecting very large amounts of data to ai.

The second thing you need. The second condition to build a big database company is that you need some new underlying change in the storage architecture that is not possible from the databases that have come before you. If you look at Snowflake and Databricks, right, commoditized, like massive fleet of HDDs, like that was not possible in it.

It just wasn’t in the air in the nineties, right? So you just didn’t, we just didn’t build these systems. S3 and and and so on was not around. And I think the architecture that is now possible that wasn’t possible 15 years ago is to go all in on NVME SSDs. It requires a particular type of architecture for the database that.

It’s difficult to retrofit onto the databases that are already there, including the ones you just mentioned. The second thing is to go all in on OIC storage, more so than we could have done 15 years ago. Like we don’t have a consensus layer, we don’t really have anything. In fact, you could turn off all the servers that Turbo Buffer has, and we would not lose any data because we have all completely all in on OIC storage.

And this means that our architecture is just so simple. So that’s the second condition, right? First being a new workload. That means that every company on earth, either indirectly or directly, is using your database. Second being, there’s some new storage architecture. That means that the, the companies that have come before you can do what you’re doing.

I think the third thing you need to do to build a big database company is that over time you have to implement more or less every Cory plan on the data. What that means is that you. You can’t just get stuck in, like, this is the one thing that a database does. It has to be ever evolving because when someone has data in the database, they over time expect to be able to ask it more or less every question.

So you have to do that to get the storage architecture to the limit of what, what it’s capable of. Those are the three conditions.

swyx: I just wanted to get a little bit of like the motivation, right? Like, so you left Shopify, you’re like principal, engineer, infra guy. Um, you also head of kernel labs, uh, inside of Shopify, right?

And then you consulted for read wise and that it kind of gave you that, that idea. I just wanted you to tell that story. Um, maybe I, you’ve told it before, but, uh, just introduce the, the. People to like the, the new workload, the sort of aha moment for turbo Puffer

Simon Hørup Eskildsen: For sure. So yeah, I spent almost a decade at Shopify.

I was on the infrastructure team, um, from the fairly, fairly early days around 2013. Um, at the time it felt like it was growing so quickly and everything, all the metrics were, you know, doubling year on year compared to the, what companies are contending with today. It’s very cute in growth. I feel like lot some companies are seeing that month over month.

Um, of course. Shopify compound has been compounding for a very long time now, but I spent a decade doing that and the majority of that was just make sure the site is up today and make sure it’s up a year from now. And a lot of that was really just the, um, you know, uh, the Kardashians would drive very, very large amounts of, of data to, to uh, to Shopify as they were rotating through all the merch and building out their businesses.

And we just needed to make sure we could handle that. Right. And sometimes these were events, a million requests per second. And so, you know, we, we had our own data centers back in the day and we were moving to the cloud and there was so much sharding work and all of that that we were doing. So I spent a decade just scaling databases ‘cause that’s fundamentally what’s the most difficult thing to scale about these sites.

The database that was the most difficult for me to scale during that time, and that was the most aggravating to be on call for, was elastic search. It was very, very difficult to deal with. And I saw a lot of projects that were just being held back in their ambition by using it.

swyx: And I mean, self-hosted.

Self-hosted. ‘cause

Simon Hørup Eskildsen: it’s, yeah, and it commercial, this is like 2015, right? So it’s like a very particular vintage. Right. It’s probably better at a lot of these things now. Um, it was difficult to contend with and I’m just like, I just think about it. It’s an inverted index. It should be good at these kinds of queries and do all of this.

And it was, we, we often couldn’t get it to do exactly what we needed to do or basically get lucine to do, like expose lucine raw to, to, to what we needed to do. Um, so that was like. Just something that we did on the side and just panic scaled when we needed to, but not a particular focus of mine. So I left, and when I left, I, um, wasn’t sure exactly what I wanted to do.

I mean, it spent like a decade inside of the same company. I’d like grown up there. I started working there when I was 18.

swyx: You only do Rails?

Simon Hørup Eskildsen: Yeah. I mean, yeah. Rails. And he’s a Rails guy. Uh, love Rails. So good. Um,

Alessio: we all wish we could still work in Rails.

swyx: I know know. I know, but some, I tried learning Ruby.

It’s just too much, like too many options to do the same thing. It’s, that’s my, I I know there’s a, there’s a way to do it.

Simon Hørup Eskildsen: I love it. I don’t know that I would use it now, like given cloud code and, and, and cursor and everything, but, um, um, but still it, like if I’m just sitting down and writing a teal code, that’s how I think.

But anyway, I left and I wasn’t, I talked to a couple companies and I was like, I don’t. I need to see a little bit more of the world here to know what I’m gonna like focus on next. Um, and so what I decided is like I was gonna, I called it like angel engineering, where I just hopped around in my friend’s companies in three months increments and just helped them out with something.

Right. And, and just vested a bit of equity and solved some interesting infrastructure problem. So I worked with a bunch of companies at the time, um, read Wise was one of them. Replicate was one of them. Um, causal, I dunno if you’ve tried this, it’s like a, it’s a spreadsheet engine Yeah. Where you can do distribution.

They sold recently. Yeah. Um, we’ve been, we used that in fp and a at, um, at Turbo Puffer. Um, so a bunch of companies like this and it was super fun. And so we’re the Chachi bt moment happened, I was with. With read Wise for a stint, we were preparing for the reader launch, right? Which is where you, you cue articles and read them later.

And I was just getting their Postgres up to snuff, like, which basically boils down to tuning, auto vacuum. So I was doing that and then this happened and we were like, oh, maybe we should build a little recommendation engine and some features to try to hook in the lms. They were not that good yet, but it was clear there was something there.

And so I built a small recommendation engine just, okay, let’s take the articles that you’ve recently read, right? Like embed all the articles and then do recommendations. It was good enough that when I ran it on one of the co-founders of Rey’s, like I found out that I got articles about, about having a child.

I’m like, oh my God, I didn’t, I, I didn’t know that, that they were having a child. I wasn’t sure what to do with that information, but the recommendation engine was good enough that it was suggesting articles, um, about that. And so there was, there was recommendations and uh, it actually worked really well.

But this was a company that was spending maybe five grand a month in total on all their infrastructure and. When I did the napkin math on running the embeddings of all the articles, putting them into a vector index, putting it in prod, it’s gonna be like 30 grand a month. That just wasn’t tenable. Right?

Like Read Wise is a proudly bootstrapped company and it’s paying 30 grand for infrastructure for one feature versus five. It just wasn’t tenable. So sort of in the bucket of this is useful, it’s pretty good, but let us, let’s return to it when the costs come down.

swyx: Did you say it grows by feature? So for five to 30 is by the number of, like, what’s the, what’s the Scaling factor scale?

It scales by the number of articles that you embed.

Simon Hørup Eskildsen: It does, but what I meant by that is like five grand for like all of the other, like the Heroku, dinos, Postgres, like all the other, and this then storage is 30. Yeah. And then like 30 grand for one feature. Right. Which is like, what other articles are related to this one.

Um, so it was just too much right to, to power everything. Their budget would’ve been maybe a few thousand dollars, which still would’ve been a lot. And so we put it in a bucket of, okay, we’re gonna do that later. We’ll wait, we will wait for the cost to come down. And that haunted me. I couldn’t stop thinking about it.

I was like, okay, there’s clearly some latent demand here. If the cost had been a 10th, we would’ve shipped it and. This was really the only data point that I had. Right. I didn’t, I, I didn’t, I didn’t go out and talk to anyone else. It was just so I started reading Right. I couldn’t, I couldn’t help myself.

Like I didn’t know what like a vector index is. I, I generally barely do about how to generate the vectors. There was a lot of hype about, this is a early 2023. There was a lot of hype about vector databases. There were raising a lot of money and it’s like, I really didn’t know anything about it. It’s like, you know, trying these little models, fine tuning them.

Like I was just trying to get sort of a lay of the land. So I just sat down. I have this. A GitHub repository called Napkin Math. And on napkin math, there’s just, um, rows of like, oh, this is how much bandwidth. Like this is how many, you know, you can do 25 gigabytes per second on average to dram. You can do, you know, five gigabytes per second of rights to an SSD, blah blah.

All of these numbers, right? And S3, how many you could do per, how much bandwidth can you drive per connection? I was just sitting down, I was like, why hasn’t anyone build a database where you just put everything on O storage and then you puff it into NVME when you use the data and you puff it into dram if you’re, if you’re querying it alive, it’s just like, this seems fairly obvious and you, the only real downside to that is that if you go all in on o storage, every right will take a couple hundred milliseconds of latency, but from there it’s really all upside, right?

You do the first go, it takes half a second. And it sort of occurred to me as like, well. The architecture is really good for that. It’s really good for AB storage, it’s really good for nvm ESSD. It’s, well, you just couldn’t have done that 10 years ago. Back to what we were talking about before. You really have to build a database where you have as few round trips as possible, right?

This is how CPUs work today. It’s how NVM E SSDs work. It’s how as, um, as three works that you want to have a very large amount of outstanding requests, right? Like basically go to S3, do like that thousand requests to ask for data in one round trip. Wait for that. Get that, like, make a new decision. Do it again, and try to do that maybe a maximum of three times.

But no databases were designed that way within NVME as is ds. You can drive like within, you know, within a very low multiple of DRAM bandwidth if you use it that way. And same with S3, right? You can fully max out the network card, which generally is not maxed out. You get very, like, very, very good bandwidth.

And, but no one had built a database like that. So I was like, okay, well can’t you just, you know, take all the vectors right? And plot them in the proverbial coordinate system. Get the clusters, put a file on S3 called clusters, do json, and then put another file for every cluster, you know, cluster one, do js O cluster two, do js ON you know that like it’s two round trips, right?

So you get the clusters, you find the closest clusters, and then you download the cluster files like the, the closest end. And you could do this in two round trips.

swyx: You were nearest neighbors locally.

Simon Hørup Eskildsen: Yes. Yes. And then, and you would build this, this file, right? It’s just like ultra simplistic, but it’s not a far shot from what the first version of Turbo Buffer was.

Why hasn’t anyone done that

Alessio: in that moment? From a workload perspective, you’re thinking this is gonna be like a read heavy thing because they’re doing recommend. Like is the fact that like writes are so expensive now? Oh, with ai you’re actually not writing that much.

Simon Hørup Eskildsen: At that point I hadn’t really thought too much about, well no actually it was always clear to me that there was gonna be a lot of rights because at Shopify, the search clusters were doing, you know, I don’t know, tens or hundreds of crew QPS, right?

‘cause you just have to have a human sit and type in. But we did, you know, I don’t know how many updates there were per second. I’m sure it was in the millions, right into the cluster. So I always knew there was like a 10 to 100 ratio on the read write. In the read wise use case. It’s, um, even, even in the read wise use case, there’d probably be a lot fewer reads than writes, right?

There’s just a lot of churn on the amount of stuff that was going through versus the amount of queries. Um, I wasn’t thinking too much about that. I was mostly just thinking about what’s the fundamentally cheapest way to build a database in the cloud today using the primitives that you have available.

And this is it, right? You just, now you have one machine and you know, let’s say you have a terabyte of data in S3, you paid the $200 a month for that, and then maybe five to 10% of that data and needs to be an NV ME SSDs and less than that in dram. Well. You’re paying very, very little to inflate the data.

swyx: By the way, when you say no one else has done that, uh, would you consider Neon, uh, to be on a similar path in terms of being sort of S3 first and, uh, separating the compute and storage?

Simon Hørup Eskildsen: Yeah, I think what I meant with that is, uh, just build a completely new database. I don’t know if we were the first, like it was very much, it was, I mean, I, I hadn’t, I just looked at the napkin math and was like, this seems really obvious.

So I’m sure like a hundred people came up with it at the same time. Like the light bulb and every invention ever. Right. It was just in the air. I think Neon Neon was, was first to it. And they’re trying, they’re retrofitted onto Postgres, right? And then they built this whole architecture where you have, you have it in memory and then you sort of.

You know, m map back to S3. And I think that was very novel at the time to do it for, for all LTP, but I hadn’t seen a database that was truly all in, right. Not retrofitting it. The database felt built purely for this no consensus layer. Even using compare and swap on optic storage to do consensus. I hadn’t seen anyone go that all in.

And I, I mean, there, there, I’m sure there was someone that did that before us. I don’t know. I was just looking at the napkin math

swyx: and, and when you say consensus layer, uh, are you strongly relying on S3 Strong consistency? You are. Okay.

So

Simon Hørup Eskildsen: that is your consensus layer. It, it is the consistency layer. And I think also, like, this is something that most people don’t realize, but S3 only became consistent in December of 2020.

swyx: I remember this coming out during COVID and like people were like, oh, like, it was like, uh, it was just like a free upgrade.

Simon Hørup Eskildsen: Yeah.

swyx: They were just, they just announced it. We saw consistency guys and like, okay, cool.

Simon Hørup Eskildsen: And I’m sure that they just, they probably had it in prod for a while and they’re just like, it’s done right.

And people were like, okay, cool. But. That’s a big moment, right? Like nv, ME SSDs, were also not in the cloud until around 2017, right? So you just sort of had like 2017 nv, ME SSDs, and people were like, okay, cool. There’s like one skew that does this, whatever, right? Takes a few years. And then the second thing is like S3 becomes consistent in 2020.

So now it means you don’t have to have this like big foundation DB or like zookeeper or whatever sitting there contending with the keys, which is how. You know, that’s what Snowflake and others have do so much

swyx: for gone

Simon Hørup Eskildsen: Exactly. Just gone. Right? And so just push to the, you know, whatever, how many hundreds of people they have working on S3 solved and then compare and swap was not in S3 at this point in time,

swyx: by the way.

Uh, I don’t know what that is, so maybe you wanna explain. Yes. Yeah.

Simon Hørup Eskildsen: Yes. So, um, what Compare and swap is, is basically, you can imagine that if you have a database, it might be really nice to have a file called metadata json. And metadata JSON could say things like, Hey, these keys are here and this file means that, and there’s lots of metadata that you have to operate in the database, right?

But that’s the simplest way to do it. So now you have might, you might have a lot of servers that wanna change the metadata. They might have written a file and want the metadata to contain that file. But you have a hundred nodes that are trying to contend with this metadata that JSON well, what compare and Swap allows you to do is basically just you download the file, you make the modifications, and then you write it only if it hasn’t changed.

While you did the modification and if not you retry. Right? Should just have this retry loops. Now you can imagine if you have a hundred nodes doing that, it’s gonna be really slow, but it will converge over time. That primitive was not available in S3. It wasn’t available in S3 until late 2024, but it was available in GCP.

The real story of this is certainly not that I sat down and like bake brained it. I was like, okay, we’re gonna start on GCS S3 is gonna get it later. Like it was really not that we started, we got really lucky, like we started on GCP and we started on GCP because tur um, Shopify ran on GCP. And so that was the platform I was most available with.

Right. Um, and I knew the Canadian team there ‘cause I’d worked with them at Shopify and so it was natural for us to start there. And so when we started building the database, we’re like, oh yeah, we have to build a, we really thought we had to build a consensus layer, like have a zookeeper or something to do this.

But then we discovered the compare and swap. It’s like, oh, we can kick the can. Like we’ll just do metadata r json and just, it’s fine. It’s probably fine. Um, and we just kept kicking the can until we had very, very strong conviction in the idea. Um, and then we kind of just hinged the company on the fact that S3 probably was gonna get this, it started getting really painful in like mid 2024.

‘cause we were closing deals with, um, um, notion actually that was running in AWS and we’re like, trust us. You, you really want us to run this in GCP? And they’re like, no, I don’t know about that. Like, we’re running everything in AWS and the latency across the cloud were so big and we had so much conviction that we bought like, you know, dark fiber between the AWS regions in, in Oregon, like in the InterExchange and GCP is like, we’ve never seen a startup like do like, what’s going on here?

And we’re just like, no, we don’t wanna do this. We were tuning like TCP windows, like everything to get the latency down ‘cause we had so high conviction in not doing like a, a metadata layer on S3. So those were the three conditions, right? Compare and swap. To do metadata, which wasn’t in S3 until late 2024 S3 being consistent, which didn’t happen until December, 2020.

Uh, 2020. And then NVMe ssd, which didn’t end in the cloud until 2017.

swyx: I mean, in some ways, like a very big like cloud success story that like you were able to like, uh, put this all together, but also doing things like doing, uh, bind our favor. That that actually is something I’ve never heard.

Simon Hørup Eskildsen: I mean, it’s very common when you’re a big company, right?

You’re like connecting your own like data center or whatever. But it’s like, it was uniquely just a pain with notion because the, um, the org, like most of the, like if you’re buying in Ashburn, Virginia, right? Like US East, the Google, like the GCP and, and AWS data centers are like within a millisecond on, on each other, on the public exchanges.

But in Oregon uniquely, the GCP data center sits like a couple hundred kilometers, like east of Portland and the AWS region sits in Portland, but the network exchange they go through is through Seattle. So it’s like a full, like 14 milliseconds or something like that. And so anyway, yeah. It’s, it’s, so we were like, okay, we can’t, we have to go through an exchange in Portland.

Yeah. And

swyx: you’d rather do this than like run your zookeeper and like

Simon Hørup Eskildsen: Yes. Way rather. It doesn’t have state, I don’t want state and two systems. Um, and I think all that is just informed by Justine, my co-founder and I had just been on call for so long. And the worst outages are the ones where you have state in multiple places that’s not syncing up.

So it really came from, from a a, like just a, a very pure source of pain, of just imagining what we would be Okay. Being woken up at 3:00 AM about and having something in zookeeper was not one of them.

swyx: You, you’re talking to like a notion or something. Do they care or do they just, they

Simon Hørup Eskildsen: just, they care about latency.

swyx: They latency cost. That’s it.

Simon Hørup Eskildsen: They just cared about latency. Right. And we just absorbed the cost. We’re just like, we have high conviction in this. At some point we can move them to AWS. Right. And so we just, we, we’ll buy the fiber, it doesn’t matter. Right. Um, and it’s like $5,000. Usually when you buy fiber, you buy like multiple lines.

And we’re like, we can only afford one, but we will just test it that when it goes over the public internet, it’s like super smooth. And so we did a lot of, anyway, it’s, yeah, it was, that’s cool.

Alessio: You can imagine talking to the GCP rep and it’s like, no, we’re gonna buy, because we know we’re gonna turn, we’re gonna turn from you guys and go to AWS in like six months.

But in the meantime we’ll do this. It’s

Simon Hørup Eskildsen: a, I mean, like they, you know, this workload still runs on GCP for what it’s worth. Right? ‘cause it’s so, it was just, it was so reliable. So it was never about moving off GCP, it was just about honesty. It was just about giving notion the latency that they deserved.

Right. Um, and we didn’t want ‘em to have to care about any of this. We also, they were like, oh, egress is gonna be bad. It was like, okay, screw it. Like we’re just gonna like vvc, VPC peer with you and AWS we’ll eat the cost. Yeah. Whatever needs to be done.

Alessio: And what were the actual workloads? Because I think when you think about ai, it’s like 14 milliseconds.

It’s like really doesn’t really matter in the scheme of like a model generation.

Simon Hørup Eskildsen: Yeah. We were told the latency, right. That we had to beat. Oh, right. So, so we’re just looking at the traces. Right. And then sort of like hand draw, like, you know, kind of like looking at the trace and then thinking what are the other extensions of the trace?

Right. And there’s a lot more to it because it’s also when you have, if you have 14 versus seven milliseconds, right. You can fit in another round trip. So we had to tune TCP to try to send as much data in every round trip, prewarm all the connections. And there was, there’s a lot of things that compound from having these kinds of round trips, but in the grand scheme it was just like, well, we have to beat the latency of whatever we’re up against.

swyx: Which is like they, I mean, notion is a database company. They could have done this themselves. They, they do lots of database engineering themselves. How do you even get in the door? Like Yeah, just like talk through that kind of.

Simon Hørup Eskildsen: Last time I was in San Francisco, I was talking to one of the engineers actually, who, who was one of our champions, um, at, AT Notion.

And they were, they were just trying to make sure that the, you know, per user cost matched the economics that they needed. You know, Uhhuh like, it’s like the way I think about, it’s like I have to earn a return on whatever the clouds charge me and then my customers have to earn a return on that. And it’s like very simple, right?

And so there has to be gross margin all the way up and that’s how you build the product. And so then our customers have to make the right set of trade off the turbo Puffer makes, and if they’re happy with that, that’s great.

swyx: Do you feel like you’re competing with build internally versus buy or buy versus buy?

Simon Hørup Eskildsen: Yeah, so, sorry, this was all to build up to your question. So one of the notion engineers told me that they’d sat and probably on a napkin, like drawn out like, why hasn’t anyone built this? And then they saw terrible. It was like, well, it literally that. So, and I think AI has also changed the buy versus build equation in terms of, it’s not really about can we build it, it’s about do we have time to build it?

I think they like, I think they felt like, okay, if this is a team that can do that and they, they feel enough like an extension of our team, well then we can go a lot faster, which would be very, very good for them. And I mean, they put us through the, through the test, right? Like we had some very, very long nights to to, to do that POC.

And they were really our biggest, our second big customer off the cursor, which also was a lot of late nights. Right.

swyx: Yeah. That, I mean, should we go into that story? The, the, the sort of Chris’s story, like a lot, um, they credit you a lot for. Working very closely with them. So I just wanna hear, I’ve heard this, uh, story from Sole’s point of view, but like, I’m curious what, what it looks like from your side.

Simon Hørup Eskildsen: I actually haven’t heard it from Sole’s point of view, so maybe you can now cross reference it. The way that I remember it was that, um, the day after we launched, which was just, you know, I’d worked the whole summer on, on the first version. Justine wasn’t part of it yet. ‘cause I just, I didn’t tell anyone that summer that I was working on this.

I was just locked in on building it because it’s very easy otherwise to confuse talking about something to actually doing it. And so I was just like, I’m not gonna do that. I’m just gonna do the thing. I launched it and at this point turbo puffer is like a rust binary running on a single eight core machine in a T Marks instance.

And me deploying it was like looking at the request log and then like command seeing it or like control seeing it to just like, okay, there’s no request. Let’s upgrade the binary. Like it was like literally the, the, the, the scrappiest thing. You could imagine it was on purpose because just like at Shopify, we did that all the time.

Like, we like move, like we ran things in tux all the time to begin with. Before something had like, at least the inkling of PMF, it was like, okay, is anyone gonna hear about this? Um, and one of the cursor co-founders Arvid reached out and he just, you know, the, the cursor team are like all I-O-I-I-M-O like, um, contenders, right?

So they just speak in bullet points and, and facts. It was like this amazing email exchange just of, this is how many QPS we have, this is what we’re paying, this is where we’re going, blah, blah, blah. And so we’re just conversing in bullet points. And I tried to get a call with them a few times, but they were, so, they were like really writing the PMF bowl here, just like late 2023.

And one time Swally emails me at like five. What was it like 4:00 AM Pacific time saying like, Hey, are you open for a call now? And I’m on the East coast and I, it was like 7:00 AM I was like, yeah, great, sure, whatever. Um, and we just started talking and something. Then I didn’t know anything about sales.

It was something that just comp compelled me. I have to go see this team. Like, there’s something here. So I, I went to San Francisco and I went to their office and the way that I remember it is that Postgres was down when I showed up at the office. Did SW tell you this? No. Okay. So Postgres was down and so it’s like they were distracting with that.

And I was trying my best to see if I could, if I could help in any way. Like I knew a little bit about databases back to tuning, auto vacuum. It was like, I think you have to tune out a vacuum. Um, and so we, we talked about that and then, um, that evening just talked about like what would it look like, what would it look like to work with us?

And I just said. Look like we’re all in, like we will just do what we’ll do whatever, whatever you tell us, right? They migrated everything over the next like week or two, and we reduced their cost by 95%, which I think like kind of fixed their per user economics. Um, and it solved a lot of other things. And we were just, Justine, this is also when I asked Justine to come on as my co-founder, she was the best engineer, um, that I ever worked with at Shopify.

She lived two blocks away and we were just, okay, we’re just gonna get this done. Um, and we did, and so we helped them migrate and we just worked like hell over the next like month or two to make sure that we were never an issue. And that was, that was the cursor story. Yeah.

swyx: And, and is code a different workload than normal text?

I, I don’t know. Is is it just text? Is it the same thing?

Simon Hørup Eskildsen: Yeah, so cursor’s workload is basically, they, um, they will embed the entire code base, right? So they, they will like chunk it up in whatever they would, they do. They have their own embedding model, um, which they’ve been public about. Um, and they find that on, on, on their evals.

It. There’s one of their evals where it’s like a 25% improvement on a very particular workload. They have a bunch of blog posts about it. Um, I think it works best on larger code basis, but they’ve trained their own embedding model to do this. Um, and so you’ll see it if you use the cursor agent, it will do searches.

And they’ve also been public around, um, how they’ve, I think they post trained their model to be very good at semantic search as well. Um, and that’s, that’s how they use it. And so it’s very good at, like, can you find me on the code that’s similar to this, or code that does this? And just in, in this queries, they also use GR to supplement it.

swyx: Yeah.

Simon Hørup Eskildsen: Um, of course

swyx: it’s been a big topic of discussion like, is rag dead because gr you know,

Simon Hørup Eskildsen: and I mean like, I just, we, we see lots of demand from the coding company to ethics

swyx: search in every part. Yes.

Simon Hørup Eskildsen: Uh, we, we, we see demand. And so, I mean, I’m. I like case studies. I don’t like, like just doing like thought pieces on this is where it’s going.

And like trying to be all macroeconomic about ai, that’s has turned out to be a giant waste of time because no one can really predict any of this. So I just collect case studies and I mean, cursor has done a great job talking about what they’re doing and I hope some of the other coding labs that use Turbo Puffer will do the same.

Um, but it does seem to make a difference for particular queries. Um, I mean we can also do text, we can also do RegX, but I should also say that cursors like security posture into Tur Puffer is exceptional, right? They have their own embedding model, which makes it very difficult to reverse engineer. They obfuscate the file paths.

They like you. It’s very difficult to learn anything about a code base by looking at it. And the other thing they do too is that for their customers, they encrypt it with their encryption keys in turbo puffer’s bucket. Um, so it’s, it’s, it’s really, really well designed.

swyx: And so this is like extra stuff they did to work with you because you are not part of Cursor.

Exactly like, and this is just best practice when working in any database, not just you guys. Okay. Yeah, that makes sense. Yeah. I think for me, like the, the, the learning is kind of like you, like all workloads are hybrid. Like, you know, uh, like you, you want the semantic, you want the text, you want the RegX, you want sql.

I dunno. Um, but like, it’s silly to like be all in on like one particularly query pattern.

Simon Hørup Eskildsen: I think, like I really like the way that, um, um, that swally at cursor talks about it, which is, um, I’m gonna butcher it here. Um, and you know, I’m a, I’m a database scalability person. I’m not a, I, I dunno anything about training models other than, um, what the internet tells me and what.

The way he describes is that this is just like cash compute, right? It’s like you have a point in time where you’re looking at some particular context and focused on some chunk and you say, this is the layer of the neural net at this point in time. That seems fundamentally really useful to do cash compute like that.

And, um, how the value of that will change over time. I’m, I’m not sure, but there seems to be a lot of value in that.

Alessio: Maybe talk a bit about the evolution of the workload, because even like search, like maybe two years ago it was like one search at the start of like an LLM query to build the context. Now you have a gentech search, however you wanna call it, where like the model is both writing and changing the code and it’s searching it again later.

Yeah. What are maybe some of the new types of workloads or like changes you’ve had to make to your architecture for it?

Simon Hørup Eskildsen: I think you’re right. When I think of rag, I think of, Hey, there’s an 8,000 token, uh, context window and you better make it count. Um, and search was a way to do that now. Everything is moving towards the, just let the agent do its thing.

Right? And so back to the thing before, right? The LLM is very good at reasoning with the data, and so we’re just the tool call, right? And that’s increasingly what we see our customers doing. Um, what we’re seeing more demand from, from our customers now is to do a lot of concurrency, right? Like Notion does a ridiculous amount of queries in every round trip just because they can’t.

And I’m also now, when I use the cursor agent, I also see them doing more concurrency than I’ve ever seen before. So a bit similar to how we designed a database to drive as much concurrency in every round trip as possible. That’s also what the agents are doing. So that’s new. It means just an enormous amount of queries all at once to the dataset while it’s warm in as few turns as possible.

swyx: Can I clarify one thing on that?

Simon Hørup Eskildsen: Yes.

swyx: Is it, are they batching multiple users or one user is driving multiple,

Simon Hørup Eskildsen: one user driving multiple, one agent driving.

swyx: It’s parallel searching a bunch of things.

Simon Hørup Eskildsen: Exactly.

swyx: Yeah. Yeah, exactly. So yeah, the clinician also did, did this for the fast context thing, like eight parallel at once.

Simon Hørup Eskildsen: Yes.

swyx: And, and like an interesting problem is, well, how do you make sure you have enough diversity so you’re not making the the same request eight times?

Simon Hørup Eskildsen: And I think like that’s probably also where the hybrid comes in, where. That’s another way to diversify. It’s a completely different way to, to do the search.

That’s a big change, right? So before it was really just like one call and then, you know, the LLM took however many seconds to return, but now we just see an enormous amount of queries. So the, um, we just see more queries. So we’ve like tried to reduce query, we’ve reduced query pricing. Um, this is probably the first time actually I’m saying that, but the query pricing is being reduced, like five x.

Um, and we’ll probably try to reduce it even more to accommodate some of these workloads of just doing very large amounts of queries. Um, that’s one thing that’s changed. I think the right, the right ratio is still very high, right? Like there’s still a, an enormous amount of rights per read, but we’re starting probably to see that change if people really lean into this pattern.

Alessio: Can we talk a little bit about the pricing? I’m curious, uh, because traditionally a database would charge on storage, but now you have the token generation that is so expensive, where like the actual. Value of like a good search query is like much higher because they’re like saving inference time down the line.

How do you structure that as like, what are people receptive to on the other side too?

Simon Hørup Eskildsen: Yeah. I, the, the turbo puffer pricing in the beginning was just very simple. The pricing on these on for search engines before Turbo Puffer was very server full, right? It was like, here’s the vm, here’s the per hour cost, right?

Great. And I just sat down with like a piece of paper and said like, if Turbo Puffer was like really good, this is probably what it would cost with a little bit of margin. And that was the first pricing of Turbo Puffer. And I just like sat down and I was like, okay, like this is like probably the storage amp, but whenever on a piece of paper I, it was vibe pricing.

It was very vibe price, and I got it wrong. Oh. Um, well I didn’t get it wrong, but like Turbo Puffer wasn’t at the first principle pricing, right? So when Cursor came on Turbo Puffer, it was like. Like, I didn’t know any VCs. I didn’t know, like I was just like, I don’t know, I didn’t know anything about raising money or anything like that.

I just saw that my GCP bill was, was high, was a lot higher than the cursor bill. So Justine and I was just like, well, we have to optimize it. Um, and I mean, to the chagrin now of, of it, of, of the VCs, it now means that we’re profitable because we’ve had so much pricing pressure in the beginning. Because it was running on my credit card and Justine and I had spent like, like tens of thousands of dollars on like compute bills and like spinning off the company and like very like, like bad Canadian lawyers and like things like to like get all of this done because we just like, we didn’t know.

Right. If you’re like steeped in San Francisco, you’re just like, you just know. Okay. Like you go out, raise a pre-seed round. I, I never heard a word pre-seed at this point in time.

swyx: When you had Cursor, you had Notion you, you had no funding.

Simon Hørup Eskildsen: Um, with Cursor we had no funding. Yeah. Um, by the time we had Notion Locke was, Locke was here.

Yeah. So it was really just, we vibe priced it 100% from first Principles, but it wasn’t, it, it was not performing at first principles, so we just did everything we could to optimize it in the beginning for that, so that at least we could have like a 5% margin or something. So I wasn’t freaking out because Cursor’s bill was also going like this as they were growing.

And so my liability and my credit limit was like actively like calling my bank. It was like, I need a bigger credit. Like it was, yeah. Anyway, that was the beginning. Yeah. But the pricing was, yeah, like storage rights and query. Right. And the, the pricing we have today is basically just that pricing with duct tape and spit to try to approach like, you know, like a, as a margin on the physical underlying hardware.

And we’re doing this year, you’re gonna see more and more pricing changes from us. Yeah.

swyx: And like is how much does stuff like VVC peering matter because you’re working in AWS land where egress is charged and all that, you know.

Simon Hørup Eskildsen: We probably don’t like, we have like an enterprise plan that just has like a base fee because we haven’t had time to figure out SKU pricing for all of this.

Um, but I mean, yeah, you can run turbo puffer either in SaaS, right? That’s what Cursor does. You can run it in a single tenant cluster. So it’s just you. That’s what Notion does. And then you can run it in, in, in BYOC where everything is inside the customer’s VPC, that’s what an for example, philanthropic does.

swyx: What I’m hearing is that this is probably the best CRO job for somebody who can come in and,

Simon Hørup Eskildsen: I mean,

swyx: help you with this.

Simon Hørup Eskildsen: Um, like Turbo Puffer hired, like, I don’t know what, what number this was, but we had a full-time CFO as like the 12th hire or something at Turbo Puffer, um, I think I hear are a lot of comp.

I don’t know how they do it. Like they have a hundred employees and not a CFO. It’s like having a CFO is like a running

swyx: business man. Like, you know,

Simon Hørup Eskildsen: it’s so good. Yeah, like money Mike, like he just, you know, just handles the money and a lot of the business stuff and so he came in and just hopped with a lot of the operational side of the business.

So like C-O-O-C-F-O, like somewhere in between.

swyx: Just as quick mention of Lucky, just ‘cause I’m curious, I’ve met Lock and like, he’s obviously a very good investor and now on physical intelligence, um, I call it generalist super angel, right? He invests in everything. Um, and I always wonder like, you know, is there something appealing about focusing on developer tooling, focusing on databases, going like, I’ve invested for 10 years in databases versus being like a lock where he can maybe like connect you to all the customers that you need.

Simon Hørup Eskildsen: This is an excellent question. No, no one’s asked me this. Um, why lockey? Because. There was a couple of people that we were talking to at the time and when we were raising, we were almost a little, we were like a bit distressed because one of our, one of our peers had just launched something that was very similar to Turbo Puffer.

And someone just gave me the advice at the time of just choose the person where you just feel like you can just pick up the phone and not prepare anything. And just be completely honest, and I don’t think I’ve said this publicly before, but I just called Lockey and was like local Lockie. Like if this doesn’t have PMF by the end of the year, like we’ll just like return all the money to you.

But it’s just like, I don’t really, we, Justine and I don’t wanna work on this unless it’s really working. So we want to give it the best shot this year and like we’re really gonna go for it. We’re gonna hire a bunch of people and we’re just gonna be honest with everyone. Like when I don’t know how to play a game, I just play with open cards and.

Lockey was the only person that didn’t, that didn’t freak out. He was like, I’ve never heard anyone say that before. As I said, I didn’t even know what a seed or pre-seed round was like before, probably even at this time. So I was just like very honest with him. And I asked him like, Lockie, have you ever have, have you ever invested in database company?

He was just like, no. And at the time I was like, am I dumb? Like, but I think there was something that just like really drew me to Lockie. He is so authentic, so honest, like, and there was something just like, I just felt like I could just play like, just say everything openly. And that was, that was, I think that that was like a perfect match at the time, and, and, and honestly still is.

He was just like, okay, that’s great. This is like the most honest, ridiculous thing I’ve ever heard anyone say to me. But like that, like that, why

swyx: is this ridiculous? Say competitor launch, this may not work out. It was

Simon Hørup Eskildsen: more just like. If this doesn’t work out, I’m gonna close up shop by the end of the mo the year, right?

Like it was, I don’t know, maybe it’s common. I, I don’t know. He told me it was uncommon. I don’t know. Um, that’s why we chose him and he’d been phenomenal. The other people were talking at the, at the time were database experts. Like they, you know, knew a lot about databases and Locke didn’t, this turned out to be a phenomenal asset.

Right. I like Justine and I know a lot about databases. The people that we hire know a lot about databases. What we needed was just someone who didn’t know a lot about databases, didn’t pretend to know a lot about databases, and just wanted to help us with candidates and customers. And he did. Yeah. And I have a list, right, of the investors that I have a relationship with, and Lockey has just performed excellent in the number of sub bullets of what we can attribute back to him.

Just absolutely incredible. And when people talk about like no ego and just the best thing for the founder, I like, I don’t think that anyone, like even my lawyer is like, yeah, Lockey is like the most friendly person you will find.

swyx: Okay. This is my most glow recommendation I’ve ever heard.

Alessio: He deserves it.

He’s very special.

swyx: Yeah. Yeah. Yeah. Okay. Amazing.

Alessio: Since you mentioned candidates, maybe we can talk about team building, you know, like, especially in sf, it feels like it’s just easier to start a company than to join a company. Uh, I’m curious your experience, especially not being n SF full-time and doing something that is maybe, you know, a very low level of detail and technical detail.

Simon Hørup Eskildsen: Yeah. So joining versus starting, I never thought that I would be a founder. I would start with it, like Turbo Puffer started as a blog post, and then it became a project and then sort of almost accidentally became a company. And now it feels like it’s, it’s like becoming a bigger company. That was never the intention.

The intentions were very pure. It’s just like, why hasn’t anyone done this? And it’s like, I wanna be the, like, I wanna be the first person to do it. I think some founders have this, like, I could never work for anyone else. I, I really don’t feel that way. Like, it’s just like, I wanna see this happen. And I wanna see it happen with some people that I really enjoy working with and I wanna have fun doing it and this, this, this has all felt very natural on that, on that sense.

So it was never a like join versus versus versus found. It was just dis found me at the right moment.

Alessio: Well I think there’s an argument for, you should have joined Cursor, right? So I’m curious like how you evaluate it. Okay, I should actually go raise money and make this a company versus like, this is like a company that is like growing like crazy.

It’s like an interesting technical problem. I should just build it within Cursor and then they don’t have to encrypt all this stuff. They don’t have to obfuscate things. Like was that on your mind at all or

Simon Hørup Eskildsen: before taking the, the small check from Lockie, I did have like a hard like look at myself in the mirror of like, okay, do I really want to do this?

And because if I take the money, I really have to do it right. And so the way I almost think about it’s like you kind of need to ha like you kind of need to be like fucked up enough to want to go all the way. And that was the conversation where I was like, okay, this is gonna be part of my life’s journey to build this company and do it in the best way that I possibly can’t.

Because if I ask people to join me, ask people to get on the cap table, then I have an ultimate responsibility to give it everything. And I don’t, I think some people, it doesn’t occur to me that everyone takes it that seriously. And maybe I take it too seriously, I don’t know. But that was like a very intentional moment.

And so then it was very clear like, okay, I’m gonna do this and I’m gonna give it everything.

Alessio: A lot of people don’t take it this seriously. But,

swyx: uh, let’s talk about, you have this concept of the P 99 engineer. Uh, people are 10 x saying, everyone’s saying, you know, uh, maybe engineers are out of a job. I don’t know.

But you definitely see a P 99 engineer, and I just want you to talk about it.

Simon Hørup Eskildsen: Yeah, so the P 99 engineer was just a term that we started using internally to talk about candidates and talk about how we wanted to build the company. And you know, like everyone else is, like we want a talent dense company.

And I think that’s almost become trite at this point. What I credit the cursor founders a lot with is that they just arrived there from first principles of like, we just need a talent dense, um, talent dense team. And I think I’ve seen some teams that weren’t talent dense and like seemed a counterfactual run, which if you’ve run in been in a large company, you will just see that like it’s just logically will happen at a large company.

Um, and so that was super important to me and Justine and it’s very difficult to maintain. And so we just needed, we needed wording for it. And so I have a document called Traits of the P 99 Engineer, and it’s a bullet point list. And I look at that list after every single interview that I do, and in every single recap that we do and every recap we end with.

End with, um, some version of I’m gonna reject this candidate completely regardless of what the discourse was, because I wanna see people fight for this person because the default should not be, we’re gonna hire this person. The default should be, we’re definitely not hiring this person. And you know, if everyone was like, ah, maybe throw a punch, then this is not the right.

swyx: Do, do you operate, like if there’s one cha there must have at least one champion who’s like, yes, I will put my career on, on, on the line for this. You know,

Simon Hørup Eskildsen: I think career on the line,

swyx: maybe a chair, but

Simon Hørup Eskildsen: yeah. You know, like, um, I would say so someone needs to like, have both fists up and be like, I’d fight.

Right? Yeah. Yeah. And if one person said, then, okay, let’s do it. Right?

swyx: Yeah.

Simon Hørup Eskildsen: Um. It doesn’t have to be absolutely everyone. Right? And like the interviews are always the sign that you’re checking for different attributes. And if someone is like knocking it outta the park in every single attribute, that’s, that’s fairly rare.

Um, but that’s really important. And so the traits of the P 99 engineer, there’s lots of them. There’s also the traits of the p like triple nine engineer and the quadruple nine engineer. This is like, it’s a long list.

swyx: Okay.

Simon Hørup Eskildsen: Um, I’ll give you some samples, right. Of what we, what we look for. I think that the P 99 engineer has some history of having bent, like their trajectory or something to their will.

Right? Some moment where it was just, they just, you know, made the computer do what it needed to do. There’s something like that, and it will, it will occur to have them at some point in their career. And, uh. Hopefully multiple times. Right.

swyx: Gimme an example of one of your engineers that like,

Simon Hørup Eskildsen: I’ll give an eng.

Uh, so we, we, we launched this thing called A and NV three. Um, we could, we’re also, we’re working on V four and V five right now, but a and NV three can search a hundred billion vectors with a P 50 of around 40 milliseconds and a p 99 of 200 milliseconds. Um, maybe other people have done this, I’m sure Google and others have done this, but, uh, we haven’t seen anyone, um, at least not in like a public consumable SaaS that can do this.

And that was an engineer, the chief architect of Turbo Puffer, Nathan, um, who more or less just bent this, the software was not capable of this and he just made it capable for a very particular workload in like a, you know, six to eight week period with the help of a lot of the team. Right. It’s been, been, there’s numerous of examples of that, like at, at turbo puff, but that’s like really bending the software and X 86 to your will.

It was incredible to watch. Um. You wanna see some moments like that?

swyx: Isn’t that triple nine?

Simon Hørup Eskildsen: Um, I think Nathan, what’s called

Alessio: group nine, that was only nine. I feel like this is too high for

Simon Hørup Eskildsen: Nathan. Nathan is, uh, Nathan is like, yeah, there’s a lot of nines. Okay. After that p So I think that’s one trait. I think another trait is that, uh, the P 99 spends a lot of time looking at maps.

Generally it’s their preferred ux. They just love looking at maps. You ever seen someone who just like, sits on their phone and just like, scrolls around on a map? Or did you not look at maps A lot? You guys don’t look at

swyx: maps? I guess I’m not feeling there. I don’t know, but

Simon Hørup Eskildsen: you just dis What about trains?

Do you like trains?

swyx: Uh, I mean they, not enough. Okay. This is just like weapon nice. Autism is what I call it. Like, like,

Simon Hørup Eskildsen: um, I love looking at maps, like, it’s like my preferred UX and just like I, you know, I like

swyx: lots

Alessio: of, of like random places, so

swyx: like,

you

swyx: know.

Alessio: Yes. Okay. There you go. So instead of like random places, like how do you explore the maps?

Simon Hørup Eskildsen: No, it’s, it’s just a joke.

swyx: It’s autism laugh. It’s like you are just obsessed by something and you like studying a thing.

Simon Hørup Eskildsen: The origin of this was that at some point I read an interview with some IOI gold medalist

swyx: Uhhuh,

Simon Hørup Eskildsen: and it’s like, what do you do in your spare time? I was just like, I like looking at maps.

I was like, I feel so seen. Like, I just like love, like swirling out. I was like, oh, Canada is so big. Where’s Baffin Island? I don’t know. I love it. Yeah. Um, anyway, so the traits of P 99, P 99 is obsessive, right? Like, there’s just like, you’ll, you’ll find traits of that we do an interview at, at, at, at turbo puffer or like multiple interviews that just try to screen for some of these things.

Um, so. There’s lots of others, but these are the kinds of traits that we look for.

swyx: I’ll tell you, uh, some people listen for like some of my dere stuff. Uh, I do think about derel as maps. Um, you draw a map for people, uh, maps show you the, uh, what is commonly agreed to be the geographical features of what a boundary is.

And it shows also shows you what is not doing. And I, I think a lot of like developer tools, companies try to tell you they can do everything, but like, let’s, let’s be real. Like you, your, your three landmarks are here, everyone comes here, then here, then here, and you draw a map and, and then you draw a journey through the map.

And like that. To me, that’s what developer relations looks like. So I do think about things that way.

Simon Hørup Eskildsen: I think the P 99 thinks in offs, right? The P 99 is very clear about, you know, hey, turbo puffer, you can’t run a high transaction workload on turbo puffer, right? It’s like the right latency is a hundred milliseconds.

That’s a clear trade off. I think the P 99 is very good at articulating the trade offs in every decision. Um. Which is exactly what the map is in your case, right?

swyx: Uh, yeah, yeah. My, my, my world. My world.

Alessio: How, how do you reconcile some of these things when you’re saying you bend the will the computer versus like the tradeoffs?

You know, I think sometimes it’s like, well, these are the tradeoffs, but the three nines, it’s like, actually it’s not a real trade off because we can make something that nobody has ever made before and actually make it work.

Simon Hørup Eskildsen: The way I think about the bending trajectory to your will is, um, if you sit down and do the napkin math, right, where you’re just like, okay, like if I have a hundred machines, they have this many terabytes of disc, they have this bandwidth, whatever, right?

And you sit down and you just do the like high school napkin math on this is how many qps we should be able to drive to it. Similar to how I did the vibe pricing, right? If you can sit down and do that, and then you observe the real system and you see, oh, we’re off by like 10 x bendings trajectory to your will is like just making the software get closer and closer to that first principle line.

The P 99 might even be able to cross the line Right. By finding even more optimizations than, than, than from first principle. So bending the software to your rail is about that, right? Like a hundred millisecond P 99 to um, to S3. I mean now you’re talking like someone really high agency that like goes to Seattle finds CS three team and it’s like, how are we gonna make this 10?

You know, like it’s that, that’s not quite what we talk about. Right. But yeah.

swyx: What’s the future? Turbo Puffer.

Simon Hørup Eskildsen: Turbo Puffer started out act one of Turbo Puffer was vector search. That was all we did to begin with Act two of Turbo Puffer is. Is and was full text Search Turbo Puffer today has a fairly start of the state-of-the-art full text search engine.

Um, we beat Lucine on some queries, in particular very long queries that we’ve optimized for because those are the text search queries we see today. They’re generated by LLMs or augmented by LLMs. Um, and we see them on Webscale datasets, right? Like someone searching for a very long texturing on all of Common Crawl.

We beat Lucine on some of those benchmarks and we expect to continue to beat Lucine on more and more queries. Um, that’s the performance and scale. Turbo Puffer does phenomenally now at full text search performance at scale. What we work on now is more and more features for full tech search. People expect a lot of features with full tech search.

And full tech search is still very valuable, right? If you go in and you press Command K and you search for si. Embedding based search might be like, oh, this is something agreeable. ‘cause that seat that’s yes, in Spanish, right,

Alessio: an Italian too,

Simon Hørup Eskildsen: but in full night search. That’s the prefix of maybe a document of like, you know, these are all the reasons I hate Simon, right?

Like this, this is like, that’s a completely different. So that augmentation to like how the human brain works and mapping like data to user is very important, but it’s a lot of features. That feature grind is what we’re firmly on and you will see us just adding to the change log every month, just more and more full tech search features.

Um, so we’re like fully compatible and we see we’re seeing people move from some of the traditional search engine onto Turbo Puffer, um, for that. That’s a big focus of Turbo Puffer this year. The other, the other focus of, of Turbo Puffer this year is just on scale. We’re seeing more and more companies that wanna search basically common crawl level types of data sets.

Um, both internally a companies and externally at, at a time like Cory, like a hundred billion vectors or a hundred billion documents at once. This is tricky and we wanna make it cheaper and we wanna make it faster. Um, that’s a big focus for Turbo Puffer this year. That’s, you know, we just released a NNV three, which we talked about before.

We are working on A-N-N-V-V four and we’re also have planned when we’re gonna do with a and NV five. Right. And then on full Tech search, we’re working on a lot of these features. We’ll be like FTSV three, but it will all roll out incrementally. Um, those are some of the really big features. And then the other thing is, um, our dashboard.

Have any of you ever locked into the Turbo Hover dashboard? It’s not very much there. It almost looks like if, um, a founder two years ago just sat down and wrote enough dashboard that there was at least something there, and then other people just sort of added stuff on for the next two, like the, the following two years, and then at some point SSO and other things to just catch up.

And it may or may not be have what happened, but adding like, I want PHP my admin back. Like, do you, do you guys remember? Like, it was, it was so good. Right? And I think that that like software hardware integration between the, the, the dashboard of the console of the database and the database itself. Um, I’m, I’m really excited for that.

There’s lots of other things, um, that are gonna come out in the next two. Like we talked a bit about some, some pricing and, and things like that, but those would be some of the big hitters. Right Now

swyx: you talk about eras of like turbo profile. I, I just, I have to ask like, yes, there’s the stuff that you’re working on this year, but like I’m sure in your mind you already have the next phase that you’re already thinking about

Simon Hørup Eskildsen: Act three.

swyx: Yes. Act four. Yeah.

Simon Hørup Eskildsen: Act five.

swyx: What I say about that, the candidates, you don’t have to decide. Yeah, but you know,

Simon Hørup Eskildsen: I, I’ll just say that if you wanna build a big database company, the database over time has to implement more or less every quarry plan. Because when you have your data in a database, you expect it to over time, not just search, but also, Hey, I want to aggregate this column, I want to join this data, all of that.

But when you’re a startup, your only moat is really just focus. You have to lay out the vaccine and you have to not get overeager. And I think we’ve seen some of our peers get very overeager and overextend themselves. And what I keep telling the team, I was just having breakfast this morning with our CTO and and chief architect, we were talking about like what we’re most likely to regret at the end of the year is having tried to do too much.

Um, and so Act three candidates could be, you know, a bunch of simpler ola queries, right? It could be, um, lending ourselves a little bit more into, we see some people who wanna do traces and logging and things like that. Some very simple use cases. Could be that, right? It could be maybe some time series.

Some people are trying to do that, right? Like, there’s lots of different things that you can do with turbo puffer, but for now, the, like, if you’re trying, trying to do not search on turbo puffers, the primary use case, you probably shouldn’t, but we see some customers that are like, oh, um, like at some point Cursor moved like 20 terabytes of Postgres data into Turbo Puffer because it’s like, it’s di it’s there, it works.

And these particular query plants we know work well. And so they just moved it all to defer sharding. Um, so we look for patterns like that in what future acts of turban puffer are going to be before firmly doubling down on them. But we wouldn’t, if. Today, if you’re using Turbo Puffer, it should be because search is very important to you.

And then we might do a lot of accelerated queries to that, but that should not be the main reason to go to Turbo Puffer at this point in time.

swyx: Yeah. Uh, you didn’t mention, uh, one thing I was looking for was graph type queries, like graph, database, graph, uh, queries. Can you basically trivially replicate this with what you already have?

Simon Hørup Eskildsen: We see some people doing

swyx: that, right? Because you have parallel queries and it’s It’s the same thing.

Simon Hørup Eskildsen: Exactly. So we see some people doing that, right? Like at the under, like is just a kv, right? And then we expose things on top of it. So we are seeing people do that. And I think, you know, our roadmap is very much just the database that connects AI to a very large amount of data is what the path is to do that in the right order, which is what a good startup is around what is the order to do things in.

Our customers are P 99, and they will tell us what they care most about next. And so some of them are doing graphs now, and if they need more graph database features, they’ll be banking our door and we’ll prioritize accordingly.

swyx: Tea. Okay. Give us the tea. Uh, this, you, you, uh, you kindly gifted us your favorite tea.

This is Yabu Keita Kacha, uh, from the Green Tea Shop. That’s right. Talk about your love of tea.

Simon Hørup Eskildsen: Yeah, we, we were just talking beforehand about, um, um, um. Caffeine, I think, um, and, uh, especially when I’m on a trip like this to San Francisco, I consume a lot of caffeine. Um, but this is my preferred, uh, preferred caffeine.

It’s this green tea. I have an Airtable with 200 teas that I’ve tried over time, over the past, like 15 years, and this one is my favorite. Now, when you drink a tea, um, there’s different, there’s like six different types of tea. I like green tea in particular. I generally prefer Chinese green tea. And I don’t really like Japanese green tea, but this little prefecture somewhere in Japan has specialized in like, they’re like Japanese, but doing it the Chinese way and it’s just phenomenal.

But then the interesting thing about the tea world is that all of the different, um, like you can find this particular tea, there’s probably, you know, hundreds of. Places that sell it, but they all go to a different family right on whatever mountain that they have. These like chameleon, ensis, bush bushes on and this woman, Japanese woman in Toronto from the Green tea shop, um, I don’t know, she’s just like, I has found a really good family.

‘cause that’s the best one. The best time of years to get this is in a few months when they do the spring harvest. Uh, now it’s like kind of old. Um, it’s just like, I love the spring for the fresh tea, so I hope you enjoy it. But it’s not the right time of year.

swyx: It’s out of season. Yeah. I, I, I actually didn’t even know Tea has seasons.

This is unsophisticated, but I, I think it, like, it ties in with like, you know, loving maps and being obsessed and being keen on united in everything that you do. Um, yeah. But. That’s great.

Alessio: Awesome. Well, as we were saying, we have instant hot water at Kernel. So, uh, MET lover can come by any,

Simon Hørup Eskildsen: I I have a little ticket where I bring a, uh, where I bring like a little thermometer to like a little thermoworks thermometer.

Um, last Friday when we do demos, I have this thing where if there’s not enough demos, then I fill the remaining time talking about something completely ridiculous as an incentive for people to actually demo. And last night in time, I spent 20 minutes do, walking through my air table and going through my entire tea travel kit, including the, the, the temperature monitor.

‘cause like Yeah, you’ll show up. There’s only a boiler. You can’t get it to the right. Yeah. You know, you need this at 80 degrees, but anyway. Yeah, sorry.

Alessio: Yeah, we have a, we have electric kettle with the temperature thing at home.

swyx: I would watch this. You should start a company, YouTube, but it doesn’t have anything about search.

It just has and like other brands,

Simon Hørup Eskildsen: I don’t think I could talk, but something that I started doing. Do you, um, do you two know Sam Lambert of, of course, planet Scale? Of

swyx: course.

Simon Hørup Eskildsen: Um,

swyx: very outspoken guy.

Simon Hørup Eskildsen: I love the guy and we just, um, we just last, like last week we just went on X live and just sat and like shut the s**t for like an hour and I think we’ll probably do that again.

Yes. We’ll probably come up there. Well, I don’t know what we’ll call it. Maybe P 99 live or the P 99 pod or something like that.

swyx: Um, P pod.

Simon Hørup Eskildsen: P pod.

swyx: Uh, cool. Well thank you so much for your time. I know you have to go, uh, but this is a, a blast and you’re clearly very passionate and charismatic, so, uh, I I bet you’ll get some, uh, P nine nine engineers outta this podcast.

Yeah.

Simon Hørup Eskildsen: Thank you so much for having me. It was a pleasure.

Join Kyle, Nader, Vibhu, and swyx live at NVIDIA GTC next week!

Now that AIE Europe tix are ~sold out, our attention turns to Miami and World’s Fair!

The definitive AI Accelerator chip company has more than 10xed this AI Summer:

And is now a $4.4 trillion megacorp… that is somehow still moving like a startup. We are blessed to have a unique relationship with our first ever NVIDIA guests: Kyle Kranen who gave a great inference keynote at the first World’s Fair and is one of the leading architects of NVIDIA Dynamo (a Datacenter scale inference framework supporting SGLang, TRT-LLM, vLLM), and Nader Khalil, a friend of swyx from our days in Celo in The Arena, who has been drawing developers at GTC since before they were even a glimmer in the eye of NVIDIA:

Nader discusses how NVIDIA Brev has drastically reduced the barriers to entry for developers to get a top of the line GPU up and running, and Kyle explains NVIDIA Dynamo as a data center scale inference engine that optimizes serving by scaling out, leveraging techniques like prefill/decode disaggregation, scheduling, and Kubernetes-based orchestration, framed around cost, latency, and quality tradeoffs.

We also dive into Jensen’s “SOL” (Speed of Light) first-principles urgency concept, long-context limits and model/hardware co-design, internal model APIs (https://build.nvidia.com), and upcoming Dynamo and agent sessions at GTC.

Full Video pod on YouTube

Timestamps

00:00 Agent Security Basics00:39 Podcast Welcome and Guests07:19 Acquisition and DevEx Shift13:48 SOL Culture and Dynamo Setup27:38 Why Scale Out Wins29:02 Scale Up Limits Explained30:24 From Laptop to Multi Node33:07 Cost Quality Latency Tradeoffs38:42 Disaggregation Prefill vs Decode41:05 Kubernetes Scaling with Grove43:20 Context Length and Co Design57:34 Security Meets Agents58:01 Agent Permissions Model59:10 Build Nvidia Inference Gateway01:01:52 Hackathons And Autonomy Dreams01:10:26 Local GPUs And Scaling Inference01:15:31 Long Running Agents And SF Reflections

Transcript

Agent Security Basics

Nader: Agents can do three things. They can access your files, they can access the internet, and then now they can write custom code and execute it. You literally only let an agent do two of those three things. If you can access your files and you can write custom code, you don’t want internet access because that’s one to see full vulnerability, right?

If you have access to internet and your file system, you should know the full scope of what that agent’s capable of doing. Otherwise, now we can get injected or something that can happen. And so that’s a lot of what we’ve been thinking about is like, you know, how do we both enable this because it’s clearly the future.

But then also, you know, what, what are these enforcement points that we can start to like protect?

swyx: All right.

Podcast Welcome and Guests

swyx: Welcome to the Lean Space podcast in the Chromo studio. Welcome to all the guests here. Uh, we are back with our guest host Viu. Welcome. Good to have you back. And our friends, uh, Netter and Kyle from Nvidia. Welcome.

Kyle: Yeah, thanks for having us.

swyx: Yeah, thank you. Actually, I don’t even know your titles.

Uh, I know you’re like architect something of Dynamo.

Kyle: Yeah. I, I’m one of the engineering leaders [00:01:00] and a architects of Dynamo.

swyx: And you’re director of something and developers, developer tech.

Nader: Yeah.

swyx: You’re the developers, developers, developers guy at nvidia,

Nader: open source agent marketing, brev,

swyx: and like

Nader: Devrel tools and stuff.

swyx: Yeah. Been

Nader: the focus.

swyx: And we’re, we’re kind of recording this ahead of Nvidia, GTC, which is coming to town, uh, again, uh, or taking over town, uh, which, uh, which we’ll all be at. Um, and we’ll talk a little bit about your sessions and stuff. Yeah.

Nader: We’re super excited for it.

GTC Booth Stunt Stories

swyx: One of my favorite memories for Nader, like you always do like marketing stunts and like while you were at Rev, you like had this surfboard that you like, went down to GTC with and like, NA Nvidia apparently, like did so much that they bought you.

Like what, what was that like? What was that?

Nader: Yeah. Yeah, we, we, um. Our logo was a chaka. We, we, uh, we were always just kind of like trying to keep true to who we were. I think, you know, some stuff, startups, you’re like trying to pretend that you’re a bigger, more mature company than you are. And it was actually Evan Conrad from SF Compute who was just like, you guys are like previous

swyx: guest.

Yeah.

Nader: Amazing. Oh, really? Amazing. Yeah. He was just like, guys, you’re two dudes in the room. Why are you [00:02:00] pretending that you’re not? Uh, and so then we were like, okay, let’s make the logo a shaka. We brought surfboards to our booth to GTC and the energy was great. Yeah. Some palm trees too. They,

Kyle: they actually poked out over like the, the walls so you could, you could see the bread booth.

Oh, that’s so funny. And

Nader: no one else,

Kyle: just from very far away.

Nader: Oh, so you remember it back

Kyle: then? Yeah I remember it pre-acquisition. I was like, oh, those guys look cool,

Nader: dude. That makes sense. ‘cause uh, we, so we signed up really last minute, and so we had the last booth. It was all the way in the corner. And so I was, I was worried that no one was gonna come.

So that’s why we had like the palm trees. We really came in with the surfboards. We even had one of our investors bring her dog and then she was just like walking the dog around to try to like, bring energy towards our booth. Yeah.

swyx: Steph.

Kyle: Yeah. Yeah, she’s the best,

swyx: you know, as a conference organizer, I love that.

Right? Like, it’s like everyone who sponsors a conference comes, does their booth. They’re like, we are changing the future of ai or something, some generic b******t and like, no, like actually try to stand out, make it fun, right? And people still remember it after three years.

Nader: Yeah. Yeah. You know what’s so funny?

I’ll, I’ll send, I’ll give you this clip if you wanna, if you wanna add it [00:03:00] in, but, uh, my wife was at the time fiance, she was in medical school and she came to help us. ‘cause it was like a big moment for us. And so we, we bought this cricket, it’s like a vinyl, like a vinyl, uh, printer. ‘cause like, how else are we gonna label the surfboard?

So, we got a surfboard, luckily was able to purchase that on the company card. We got a cricket and it was just like fine tuning for enterprises or something like that, that we put on the. On the surfboard and it’s 1:00 AM the day before we go to GTC. She’s helping me put these like vinyl stickers on.

And she goes, you son of, she’s like, if you pull this off, you son of a b***h. And so, uh, right. Pretty much after the acquisition, I stitched that with the mag music acquisition. I sent it to our family group chat. Oh

swyx: Yeah. No, well, she, she made a good choice there. Was that like basically the origin story for Launchable is that we, it was, and maybe we should explain what Brev is and

Nader: Yeah.

Yeah. Uh, I mean, brev is just, it’s a developer tool that makes it really easy to get a GPU. So we connect a bunch of different GPU sources. So the basics of it is like, how quickly can we SSH you into a G, into a GPU and whenever we would talk to users, they wanted A GPU. They wanted an A 100. And if you go to like any cloud [00:04:00] provisioning page, usually it’s like three pages of forms or in the forms somewhere there’s a dropdown.

And in the dropdown there’s some weird code that you know to translate to an A 100. And I remember just thinking like. Every time someone says they want an A 100, like the piece of text that they’re telling me that they want is like, stuffed away in the corner. Yeah. And so we were like, what if the biggest piece of text was what the user’s asking for?

And so when you go to Brev, it’s just big GPU chips with the type that you want with

swyx: beautiful animations that you worked on pre, like pre you can, like, now you can just prompt it. But back in the day. Yeah. Yeah. Those were handcraft, handcrafted artisanal code.

Nader: Yeah. I was actually really proud of that because, uh, it was an, i I made it in Figma.

Yeah. And then I found, I was like really struggling to figure out how to turn it from like Figma to react. So what it actually is, is just an SVG and I, I have all the styles and so when you change the chip, whether it’s like active or not it changes the SVG code and that somehow like renders like, looks like it’s animating, but it, we just had the transition slow, but it’s just like the, a JavaScript function to change the like underlying SVG.

Yeah. And that was how I ended up like figuring out how to move it from from Figma. But yeah, that’s Art Artisan. [00:05:00]

Kyle: Speaking of marketing stunts though, he actually used those SVGs. Or kind of use those SVGs to make these cards.

Nader: Oh yeah. Like

Kyle: a GPU gift card Yes. That he handed out everywhere. That was actually my first impression of that

Nader: one.

Yeah,

swyx: yeah, yeah.

Nader: Yeah.

swyx: I think I still have one of them.

Nader: They look great.

Kyle: Yeah.

Nader: I have a ton of them still actually in our garage, which just, they don’t have labels. We should honestly like bring, bring them back. But, um, I found this old printing press here, actually just around the corner on Ven ness. And it’s a third generation San Francisco shop.

And so I come in an excited startup founder trying to like, and they just have this crazy old machinery and I’m in awe. ‘cause the the whole building is so physical. Like you’re seeing these machines, they have like pedals to like move these saws and whatever. I don’t know what this machinery is, but I saw all three generations.

Like there’s like the grandpa, the father and the son, and the son was like, around my age. Well,

swyx: it’s like a holy, holy trinity.

Nader: It’s funny because we, so I just took the same SVG and we just like printed it and it’s foil printing, so they make a a, a mold. That’s like an inverse of like the A 100 and then they put the foil on it [00:06:00] and then they press it into the paper.

And I remember once we got them, he was like, Hey, don’t forget about us. You know, I guess like early Apple and Cisco’s first business cards were all made there. And so he was like, yeah, we, we get like the startup businesses but then as they mature, they kind of go somewhere else. And so I actually, I think we were talking with marketing about like using them for some, we should go back and make some cards.

swyx: Yeah, yeah, yeah. You know, I remember, you know, as a very, very small breadth investor, I was like, why are we spending time like, doing these like stunts for GPUs? Like, you know, I think like as a, you know, typical like cloud hard hardware person, you go into an AWS you pick like T five X xl, whatever, and it’s just like from a list and you look at the specs like, why animate this GP?

And, and I, I do think like it just shows the level of care that goes throughout birth and Yeah. And now, and also the, and,

Nader: and Nvidia. I think that’s what the, the thing that struck me most when we first came in was like the amount of passion that everyone has. Like, I think, um, you know, you talk to, you talk to Kyle, you talk to, like, every VP that I’ve met at Nvidia goes so close to the metal.

Like, I remember it was almost a year ago, and like my VP asked me, he’s like, Hey, [00:07:00] what’s cursor? And like, are you using it? And if so, why? Surprised at this, and he downloaded Cursor and he was asking me to help him like, use it. And I thought that was, uh, or like, just show him what he, you know, why we were using it.

And so, the amount of care that I think everyone has and the passion, appreciate, passion and appreciation for the moment. Right. This is a very unique time. So it’s really cool to see everyone really like, uh, appreciate that.

swyx: Yeah.

Acquisition and DevEx Shift

swyx: One thing I wanted to do before we move over to sort of like research topics and, uh, the, the stuff that Kyle’s working on is just tell the story of the acquisition, right?

Like, not many people have been, been through an acquisition with Nvidia. What’s it like? Uh, what, yeah, just anything you’d like to say.

Nader: It’s a crazy experience. I think, uh, you know, we were the thing that was the most exciting for us was. Our goal was just to make it easier for developers.

We wanted to find access to GPUs, make it easier to do that. And then all, oh, actually your question about launchable. So launchable was just make one click exper, like one click deploys for any software on top of the GPU. Mm-hmm. And so what we really liked about Nvidia was that it felt like we just got a lot more resources to do all of that.

I think, uh, you [00:08:00] know, NVIDIA’s goal is to make things as easy for developers as possible. So there was a really nice like synergy there. I think that, you know, when it comes to like an acquisition, I think the amount that the soul of the products align, I think is gonna be. Is going speak to the success of the acquisition.

Yeah. And so it in many ways feels like we’re home. This is a really great outcome for us. Like we you know, I love brev.nvidia.com. Like you should, you should use it’s, it’s the

Kyle: front page for GPUs.

Nader: Yeah. Yeah. If you want GP views,

Kyle: you go there, get

swyx: it there, and it’s like internally is growing very quickly.

I, I don’t remember You said some stats there.

Nader: Yeah, yeah, yeah. It’s, uh, I, I wish I had the exact numbers, but like internally, externally, it’s been growing really quickly. We’ve been working with a bunch of partners with a bunch of different customers and ISVs, if you have a solution that you want someone that runs on the GPU and you want people to use it quickly, we can bundle it up, uh, in a launchable and make it a one click run.

If you’re doing things and you want just like a sandbox or something to run on, right. Like open claw. Huge moment. Super exciting. Our, uh, and we’ll talk into it more, but. You know, internally, people wanna run this, and you, we know we have to be really careful from the security implications. Do we let this run on the corporate network?

Security’s guidance was, Hey, [00:09:00] run this on breath, it’s in, you know, it’s, it’s, it’s a vm, it’s sitting in the cloud, it’s off the corporate network. It’s isolated. And so that’s been our stance internally and externally about how to even run something like open call while we figure out how to run these things securely.

But yeah,

swyx: I think there’s also like, you almost like we’re the right team at the right time when Nvidia is starting to invest a lot more in developer experience or whatever you call it. Yeah. Uh, UX or I don’t know what you call it, like software. Like obviously NVIDIA is always invested in software, but like, there’s like, this is like a different audience.

Yeah. It’s a

Nader: wider

Kyle: developer base.

swyx: Yeah. Right.

Nader: Yeah. Yeah. You know, it’s funny, it’s like, it’s not, uh,

swyx: so like, what, what is it called internally? What, what is this that people should be aware that is going on there?

Nader: Uh, what, like developer experience

swyx: or, yeah, yeah. Is it’s called just developer experience or is there like a broader strategy here

Nader: in Nvidia?

Um, Nvidia always wants to make a good developer experience. The thing is and a lot of the technology is just really complicated. Like, it’s not, it’s uh, you know, I think, um. The thing that’s been really growing or the AI’s growing is having a huge moment, not [00:10:00] because like, let’s say data scientists in 2018, were quiet then and are much louder now.

The pie is com, right? There’s a whole bunch of new audiences. My mom’s wondering what she’s doing. My sister’s learned, like taught herself how to code. Like the, um, you know, I, I actually think just generally AI’s a big equalizer and you’re seeing a more like technologically literate society, I guess.

Like everyone’s, everyone’s learning how to code. Uh, there isn’t really an excuse for that. And so building a good UX means that you really understand who your end user is. And when your end user becomes such a wide, uh, variety of people, then you have to almost like reinvent the practice, right? Yeah. You have

Kyle: to, and actually build more developer ux, right?

Because the, there are tiers of developer base that were added. You know, the, the hackers that are building on top of open claw, right? For example, have never used gpu. They don’t know what kuda is. They, they, they just want to run something.

Nader: Yeah.

Kyle: You need new UX that is not just. Hey, you know, how do you program something in Cuda and run it?

And then, and then we built, you know, like when Deep Learning was getting big, we built, we built Torch and, and, but so recently the amount of like [00:11:00] layers that are added to that developer stack has just exploded because AI has become ubiquitous. Everyone’s using it in different ways. Yeah. It’s

Nader: moving fast in every direction.

Vertical, horizontal.

Vibhu: Yeah. You guys, you even take it down to hardware, like the DGX Spark, you know, it’s, it’s basically the same system as just throwing it up on big GPU cluster.

Nader: Yeah, yeah, yeah. It’s amazing. Blackwell.

swyx: Yeah. Uh, we saw the preview at the last year’s GTC and that was one of the better performing, uh, videos so far, and video coverage so far.

Awesome. This will beat it. Um,

Nader: that was

swyx: actually, we have fingers

Nader: crossed. Yeah.

DGX Spark and Remote Access

Nader: Even when Grace Blackwell or when, um, uh, DGX Spark was first coming out getting to be involved in that from the beginning of the developer experience. And it just comes back to what you

swyx: were involved.

Nader: Yeah. St. St.

swyx: Mars.

Nader: Yeah. Yeah. I mean from, it was just like, I, I got an email, we just got thrown into the loop and suddenly yeah, I, it was actually really funny ‘cause I’m still pretty fresh from the acquisition and I’m, I’m getting an email from a bunch of the engineering VPs about like, the new hardware, GPU chip, like we’re, or not chip, but just GPU system that we’re putting out.

And I’m like, okay, cool. Matters. Now involved with this for the ux, I’m like. What am I gonna do [00:12:00] here? So, I remember the first meeting, I was just like kind of quiet as I was hearing engineering VPs talk about what this box could be, what it could do, how we should use it. And I remember, uh, one of the first ideas that people were idea was like, oh, the first thing that it was like, I think a quote was like, the first thing someone’s gonna wanna do with this is get two of them and run a Kubernetes cluster on top of them.

And I was like, oh, I think I know why I’m here. I was like, the first thing we’re doing is easy. SSH into the machine. And then, and you know, just kind of like scoping it down of like, once you can do that every, you, like the person who wants to run a Kubernetes cluster onto Sparks has a higher propensity for pain, then, then you know someone who buys it and wants to run open Claw right now, right?

If you can make sure that that’s as effortless as possible, then the rest becomes easy. So there’s a tool called Nvidia Sync. It just makes the SSH connection really simple. So, you know, if you think about it like. If you have a Mac, uh, or a PC or whatever, if you have a laptop and you buy this GPU and you want to use it, you should be able to use it like it’s A-A-G-P-U in the cloud, right?

Um, but there’s all this friction of like, how do you actually get into that? That’s part of [00:13:00] Revs value proposition is just, you know, there’s a CLI that wraps SSH and makes it simple. And so our goal is just get you into that machine really easily. And one thing we just launched at CES, it’s in, it’s still in like early access.

We’re ironing out some kinks, but it should be ready by GTC. You can register your spark on Brev. And so now if you

swyx: like remote managed yeah, local hardware. Single pane of glass. Yeah. Yeah. Because Brev can already manage other clouds anyway, right?

Vibhu: Yeah, yeah. And you use the spark on Brev as well, right?

Nader: Yeah. But yeah, exactly. So, so you, you, so you, you set it up at home you can run the command on it, and then it gets it’s essentially it’ll appear in your Brev account, and then you can take your laptop to a Starbucks or to a cafe, and you’ll continue to use your, you can continue use your spark just like any other cloud node on Brev.

Yeah. Yeah. And it’s just like a pre-provisioned center

swyx: in your

Nader: home. Yeah, exactly.

swyx: Yeah. Yeah.

Vibhu: Tiny little data center.

Nader: Tiny little, the size of

Vibhu: your phone.

SOL Culture and Dynamo Setup

swyx: One more thing before we move on to Kyle. Just have so many Jensen stories and I just love, love mining Jensen stories. Uh, my favorite so far is SOL. Uh, what is, yeah, what is S-O-L-S-O-L

Nader: is actually, i, I think [00:14:00] of all the lessons I’ve learned, that one’s definitely my favorite.

Kyle: It’ll always stick with you.

Nader: Yeah. Yeah. I, you know, in your startup, everything’s existential, right? Like we’ve, we’ve run out of money. We were like, on the risk of, of losing payroll, we’ve had to contract our team because we l ran outta money. And so like, um, because of that you’re really always forcing yourself to I to like understand the root cause of everything.

If you get a date, if you get a timeline, you know exactly why that date or timeline is there. You’re, you’re pushing every boundary and like, you’re not just say, you’re not just accepting like a, a no. Just because. And so as you start to introduce more layers, as you start to become a much larger organization, SOL is is essentially like what is the physics, right?

The speed of light moves at a certain speed. So if flight’s moving some slower, then you know something’s in the way. So before trying to like layer reality back in of like, why can’t this be delivered at some date? Let’s just understand the physics. What is the theoretical limit to like, uh, how fast this can go?

And then start to tell me why. ‘cause otherwise people will start telling you why something can’t be done. But actually I think any great leader’s goal is just to create urgency. Yeah. [00:15:00] There’s an infinite

Kyle: create compelling events, right?

Nader: Yeah.

Kyle: Yeah. So l is a term video is used to instigate a compelling event.

You say this is done. How do we get there? What is the minimum? As much as necessary, as little as possible thing that it takes for us to get exactly here and. It helps you just break through a bunch of noise.

swyx: Yeah.

Kyle: Instantly.

swyx: One thing I’m unclear about is, can only Jensen use the SOL card? Like, oh, no, no, no.

Not everyone get the b******t out because obviously it’s Jensen, but like, can someone else be like, no, like

Kyle: frontline engineers use it.

Nader: Yeah. Every, I think it’s not so much about like, get the b******t out. It’s like, it’s like, give me the root understanding, right? Like, if you tell me something takes three weeks, it like, well, what’s the first principles?

Yeah, the first principles. It’s like, what’s the, what? Like why is it three weeks? What is the actual yeah. What’s the actual limit of why this is gonna take three weeks? If you’re gonna, if you, if let’s say you wanted to buy a new computer and someone told you it’s gonna be here in five days, what’s the SOL?

Well, like the SOL is like, I could walk into a Best Buy and pick it up for you. Right? So then anything that’s like beyond that is, and is that practical? Is that how we’re gonna, you know, let’s say give everyone in the [00:16:00] company a laptop, like obviously not. So then like that’s the SOL and then it’s like, okay, well if we have to get more than 10, suddenly there might be some, right?

And so now we can kind of piece the reality back.

swyx: So, so this is the. Paul Graham do things that don’t scale. Yeah. And this is also the, what people would now call behi agency. Yeah.

Kyle: It’s actually really interesting because there’s a, there’s a second hardware angle to SOL that like doesn’t come up for all the org sol is used like culturally at a

swyx: media for everything.

I’m also mining for like, I think that can be annoying sometimes. And like someone keeps going IOO you and you’re like, guys, like we have to be stable. We have to, we to f*****g plan. Yeah.

Kyle: It’s an interesting balance.

Nader: Yeah. I encounter that with like, actually just with, with Alec, right? ‘cause we, we have a new conference so we need to launch, we have, we have goals of what we wanna launch by, uh, by the conference and like, yeah.

At the end of the day, where is

swyx: this GTC?

Nader: Um, well this is like, so we, I mean we did it for CES, we did for GT CDC before that we’re doing it for GTC San Jose. So I mean, like every, you know, we have a new moment. Um, and we want to launch something. Yeah. And we want to do so at SOL and that does mean that some, there’s some level of prioritization that needs [00:17:00] to happen.

And so it, it is difficult, right? I think, um, you have to be careful with what you’re pushing. You know, stability is important and that should be factored into S-O-L-S-O-L isn’t just like, build everything and let it break, you know, that, that’s part of the conversation. So as you’re laying, layering in all the details, one of them might be, Hey, we could build this, but then it’s not gonna be stable for X, y, z reasons.

And so that was like, one of our conversations for CES was, you know, hey, like we, we can get this into early access registering your spark with brev. But there are a lot of things that we need to do in order to feel really comfortable from a security perspective, right? There’s a lot of networking involved before we deliver that to users.

So it’s like, okay. Let’s get this to a point where we can at least let people experiment with it. We had it in a booth, we had it in Jensen’s keynote, and then let’s go iron out all the networking kinks. And that’s not easy. And so, uh, that can come later. And so that was the way that we layered that back in.

Yeah. But

Kyle: It’s not really about saying like, you don’t have to do the, the maintenance or operational work. It’s more about saying, you know, it’s kind of like [00:18:00] highlights how progress is incremental, right? Like, what is the minimum thing that we can get to. And then there’s SOL for like every component after that.

But there’s the SOL to get you, get you to the, the starting line. And that, that’s usually how it’s asked. Yeah. On the other side, you know, like SOL came out of like hardware at Nvidia. Right. So SOL is like literally if we ran the accelerator or the GPU with like at basically full speed with like no other constraints, like how FAST would be able to make a program go.

swyx: Yeah. Yeah. Right.

Kyle: So

swyx: in, in training that like, you know, then you work back to like some percentage of like MFU for example.

Kyle: Yeah, that’s a, that’s a great example. So like, there’s an, there’s an S-O-L-M-F-U, and then there’s like, you know, what’s practically achievable.

swyx: Cool. Should we move on to sort of, uh, Kyle’s side?

Uh, Kyle, you’re coming more from the data science world. And, uh, I, I mean I always, whenever, whenever I meet someone who’s done working in tabular stuff, graph neural networks, time series, these are basically when I go to new reps, I go to ICML, I walk the back halls. There’s always like a small group of graph people.

Yes. Absolute small group of tabular people. [00:19:00] And like, there’s no one there. And like, it’s very like, you know what I mean? Like, yeah, no, like it’s, it’s important interesting work if you care about solving the problems that they solve.

Kyle: Yeah.

swyx: But everyone else is just LMS all the time.

Kyle: Yeah. I mean it’s like, it’s like the black hole, right?

Has the event horizon reached this yet in nerves? Um,

swyx: but like, you know, those are, those are transformers too. Yeah. And, and those are also like interesting things. Anyway, uh, I just wanted to spend a little bit of time on, on those, that background before we go into Dynamo, uh, proper.

Kyle: Yeah, sure. I took a different path to Nvidia than that, or I joined six years ago, seven, if you count, when I was an intern.

So I joined Nvidia, like right outta college. And the first thing I jumped into was not what I’d done in, during internship, which was like, you know, like some stuff for autonomous vehicles, like heavyweight object detection. I jumped into like, you know, something, I’m like, recommenders, this is popular. And

swyx: yeah, he did Rexi

Kyle: as well.

Yeah, Rexi. Yeah. I mean that, that was the taboo data at the time, right? You have tables of like, audience qualities and item qualities, and you’re trying to figure out like which member of [00:20:00] the audience matches which item or, or more practically which item matches which member of the audience. And at the time, really it was like we were trying to enable.

Uh, recommender, which had historically been like a little bit of a CP based workflow into something that like, ran really well in GPUs. And it’s since been done. Like there are a bunch of libraries for Axis that run on GPUs. Uh, the common models like Deeplearning recommendation model, which came outta meta and the wide and deep model, which was used or was released by Google were very accelerated by GPUs using, you know, the fast HBM on the chips, especially to do, you know, vector lookups.

But it was very interesting at the time and super, super relevant because like we were starting to get like. This explosion of feeds and things that required rec recommenders to just actively be on all the time. And sort of transitioned that a little bit towards graph neural networks when I discovered them because I was like, okay, you can actually use graphical neural networks to represent like, relationships between people, items, concepts, and that, that interested me.

So I jumped into that at [00:21:00] Nvidia and, and got really involved for like two-ish years.

swyx: Yeah. Uh, and something I learned from Brian Zaro Yeah. Is that you can just kind of choose your own path in Nvidia.

Kyle: Oh my God. Yeah.

swyx: Which is not a normal big Corp thing. Yeah. Like you, you have a lane, you stay in your lane.

Nader: I think probably the reason why I enjoy being in a, a big company, the mission is the boss probably from a startup guy. Yeah. The mission

swyx: is the boss.

Nader: Yeah. Uh, it feels like a big game of pickup basketball. Like, you know, if you play one, if you wanna play basketball, you just go up to the court and you’re like, Hey look, we’re gonna play this game and we need three.

Yeah. And you just like find your three. That’s honestly for every new initiative that’s what it feels like. Yeah.

Vibhu: It also like shows, right? Like Nvidia. Just releasing state-of-the-art stuff in every domain. Yeah. Like, okay, you expect foundation models with Nemo tron voice just randomly parakeet.

Call parakeet just comes out another one, uh, voice. The

Kyle: video voice team has always been producing.

Vibhu: Yeah. There’s always just every other domain of paper that comes out, dataset that comes out. It’s like, I mean, it also stems back to what Nvidia has to do, right? You have to make chips years before they’re actually produced.

Right? So you need to know, you need to really [00:22:00] focus. The

Kyle: design process starts like

Vibhu: exactly

Kyle: three to five years before the chip gets to the market.

Vibhu: Yeah. I, I’m curious more about what that’s like, right? So like, you have specialist teams. Is it just like, you know, people find an interest, you go in, you go deep on whatever, and that kind of feeds back into, you know, okay, we, we expect predictions.

Like the internals at Nvidia must be crazy. Right? You know? Yeah. Yeah. You know, you, you must. Not even without selling to people, you have your own predictions of where things are going. Yeah. And they’re very based, very grounded. Right?

Kyle: Yeah. It, it, it’s really interesting. So there’s like two things that I think that Amed does, which are quite interesting.

Uh, one is like, we really index into passion. There’s a big. Sort of organizational top sound push to like ensure that people are working on the things that they’re passionate about. So if someone proposes something that’s interesting, many times they can just email someone like way up the chain that they would find this relevant and say like, Hey, can I go work on this?

Nader: It’s actually like I worked at a, a big company for a couple years before, uh, starting on my startup journey and like, it felt very weird if you were to like email out of chain, if that makes [00:23:00] sense. Yeah. The emails at Nvidia are like mosh pits

swyx: shoot,

Nader: and it’s just like 60 people, just whatever. And like they’re, there’s this,

swyx: they got messy like, reply all you,

Nader: oh, it’s in, it’s insane.

It’s insane. They just

Kyle: help. You know, Maxim,

Nader: the context. But, but that’s actually like, I’ve actually, so this is a weird thing where I used to be like, why would we send emails? We have Slack. I am the entire, I’m the exact opposite. I feel so bad for anyone who’s like messaging me on Slack ‘cause I’m so unresponsive.

swyx: Your email

Nader: Maxi, email Maxim. I’m email maxing Now email is a different, email is perfect because man, we can’t work together. I’m email is great, right? Because important threads get bumped back up, right? Yeah, yeah. Um, and so Slack doesn’t do that. So I just have like this casino going off on the right or on the left and like, I don’t know which thread was from where or what, but like the threads get And then also just like the subject, so you can have like working threads.

I think what’s difficult is like when you’re small, if you’re just not 40,000 people I think Slack will work fine, but there’s, I don’t know what the inflection point is. There is gonna be a point where that becomes really messy and you’ll actually prefer having email. ‘cause you can have working threads.

You can cc more than nine people in a thread.

Kyle: You can fork stuff.

Nader: You can [00:24:00] fork stuff, which is super nice and just like y Yeah. And so, but that is part of where you can propose a plan. You can also just. Start, honestly, momentum’s the only authority, right? So like, if you can just start, start to make a little bit of progress and show someone something, and then they can try it.

That’s, I think what’s been, you know, I think the most effective way to push anything for forward. And that’s both at Nvidia and I think just generally.

Kyle: Yeah, there’s, there’s the other concept that like is explored a lot at Nvidia, which is this idea of a zero billion dollar business. Like market creation is a big thing at Nvidia.

Like,

swyx: oh, you want to go and start a zero billion dollar business?

Kyle: Jensen says, we are completely happy investing in zero billion dollar markets. We don’t care if this creates revenue. It’s important for us to know about this market. We think it will be important in the future. It can be zero billion dollars for a while.

I’m probably minging as words here for, but like, you know, like, I’ll give an example. NVIDIA’s been working on autonomous driving for a a long time,

swyx: like an Nvidia car.

Kyle: No, they, they’ve

Vibhu: used the Mercedes, right? They’re around the HQ and I think it finally just got licensed out. Now they’re starting to be used quite a [00:25:00] bit.

For 10 years you’ve been seeing Mercedes with Nvidia logos driving.

Kyle: If you’re in like the South San Santa Clara, it’s, it’s actually from South. Yeah. So, um. Zero billion dollar markets are, are a thing like, you know, Jensen,

swyx: I mean, okay, look, cars are not a zero billion dollar market. But yeah, that’s a bad example.

Nader: I think, I think he’s, he’s messaging, uh, zero today, but, or even like internally, right? Like, like it’s like, uh, an org doesn’t have to ruthlessly find revenue very quickly to justify their existence. Right. Like a lot of the important research, a lot of the important technology being developed that, that’s kind of

Kyle: where research, research is very ide ideologically free at Nvidia.

Yeah. Like they can pursue things that they were

swyx: Were you research officially?

Kyle: I was never in research. Officially. I was always in engineering. Yeah. We in, I’m in an org called Deep Warning Algorithms, which is basically just how do we make things that are relevant to deep warning go fast.

swyx: That sounds freaking cool.

Vibhu: And I think a lot of that is underappreciated, right? Like time series. This week Google put out time. FF paper. Yeah. A new time series, paper res. Uh, Symantec, ID [00:26:00] started applying Transformers LMS to Yes. Rec system. Yes. And when you think the scale of companies deploying these right. Amazon recommendations, Google web search, it’s like, it’s huge scale and

Kyle: Yeah.

Vibhu: You want fast?

Kyle: Yeah. Yeah. Yeah. Actually it’s, it, I, there’s a fun moment that brought me like full circle. Like, uh, Amazon Ads recently gave a talk where they talked about using Dynamo for generative recommendation, which was like super, like weirdly cathartic for me. I’m like, oh my God. I’ve, I’ve supplanted what I was working on.

Like, I, you’re using LMS now to do what I was doing five years ago.

swyx: Yeah. Amazing. And let’s go right into Dynamo. Uh, maybe introduce Yeah, sure. To the top down and Yeah.

Kyle: I think at this point a lot of people are familiar with the term of inference. Like funnily enough, like I went from, you know, inference being like a really niche topic to being something that’s like discussed on like normal people’s Twitter feeds.

It’s,

Nader: it’s on billboards

Kyle: here now. Yeah. Very, very strange. Driving, driving, seeing just an inference ad on 1 0 1 inference at scale is becoming a lot more important. Uh, we have these moments like, you know, open claw where you have these [00:27:00] agents that take lots and lots of tokens, but produce, incredible results.

There are many different aspects of test time scaling so that, you know, you can use more inference to generate a better result than if you were to use like a short amount of inference. There’s reasoning, there’s quiring, there’s, adding agency to the model, allowing it to call tools and use skills.

Dyno sort came about at Nvidia. Because myself and a couple others were, were sort of talking about the, these concepts that like, you know, you have inference engines like VLMS, shelan, tenor, TLM and they have like one single copy. They, they, they sort of think about like things as like one single copy, like one replica, right?

Why Scale Out Wins

Kyle: Like one version of the model. But when you’re actually serving things at scale, you can’t just scale up that replica because you end up with like performance problems. There’s a scaling limit to scaling up replicas. So you actually have to scale out to use a, maybe some Kubernetes type terminology.

We kind of realized that there was like. A lot of potential optimization that we could do in scaling out and building systems for data [00:28:00] center scale inference. So Dynamo is this data center scale inference engine that sits on top of the frameworks like VLM Shilling and 10 T lm and just makes things go faster because you can leverage the economy of scale.

The fact that you have KV cash, which we can define a little bit later, uh, in all these machines that is like unique and you wanna figure out like the ways to maximize your cash hits or you want to employ new techniques in inference like disaggregation, which Dynamo had introduced to the world in, in, in March, not introduced, it was a academic talk, but beforehand.

But we are, you know, one of the first frameworks to start, supporting it. And we wanna like, sort of combine all these techniques into sort of a modular framework that allows you to. Accelerate your inference at scale.

Nader: By the way, Kyle and I became friends on my first date, Nvidia, and I always loved, ‘cause like he always teaches me

swyx: new things.

Yeah. By the way, this is why I wanted to put two of you together. I was like, yeah, this is, this is gonna be

Kyle: good. It’s very, it’s very different, you know, like we’ve, we, we’ve, we’ve talked to each other a bunch [00:29:00] actually, you asked like, why, why can’t we scale up?

Nader: Yeah.

Scale Up Limits Explained

Nader: model, you said model replicas.

Kyle: Yeah. So you, so scale up means assigning more

swyx: heavier?

Kyle: Yeah, heavier. Like making things heavier. Yeah, adding more GPUs. Adding more CPUs. Scale out is just like having a barrier saying, I’m gonna duplicate my representation of the model or a representation of this microservice or something, and I’m gonna like, replicate it Many times.

Handle, load. And the reason that you can’t scale, scale up, uh, past some points is like, you know, there, there, there are sort of hardware bounds and algorithmic bounds on, on that type of scaling. So I’ll give you a good example that’s like very trivial. Let’s say you’re on an H 100. The Maxim ENV link domain for H 100, for most Ds H one hundreds is heus, right?

So if you scaled up past that, you’re gonna have to figure out ways to handle the fact that now for the GPUs to communicate, you have to do it over Infin band, which is still very fast, but is not as fast as ENV link.

swyx: Is it like one order of magnitude, like hundreds or,

Kyle: it’s about an order of magnitude?

Yeah. Okay. Um, so

swyx: not terrible.

Kyle: [00:30:00] Yeah. I, I need to, I need to remember the, the data sheet here, like, I think it’s like about 500 gigabytes. Uh, a second unidirectional for ENV link, and about 50 gigabytes a second unidirectional for Infin Band. I, it, it depends on the, the generation.

swyx: I just wanna set this up for people who are not familiar with these kinds of like layers and the trash speed

Vibhu: and all that.

Of course.

From Laptop to Multi Node

Vibhu: Also, maybe even just going like a few steps back before that, like most people are very familiar with. You see a, you know, you can use on your laptop, whatever these steel viol, lm you can just run inference there. All, there’s all, you can, you

can run it on that

Vibhu: laptop. You can run on laptop.

Then you get to, okay, uh, models got pretty big, right? JLM five, they doubled the size, so mm-hmm. Uh, what do you do when you have to go from, okay, I can get 128 gigs of memory. I can run it on a spark. Then you have to go multi GPU. Yeah. Okay. Multi GPU, there’s some support there. Now, if I’m a company and I don’t have like.

I’m not hiring the best researchers for this. Right. But I need to go [00:31:00] multi-node, right? I have a lot of servers. Okay, now there’s efficiency problems, right? You can have multiple eight H 100 nodes, but, you know, is that as a, like, how do you do that efficiently?

Kyle: Yeah. How do you like represent them? How do you choose how to represent the model?

Yeah, exactly right. That’s a, that’s like a hard question. Everyone asks, how do you size oh, I wanna run GLM five, which just came out new model. There have been like four of them in the past week, by the way, like a bunch of new models.

swyx: You know why? Right? Deep seek.

Kyle: No comment. Oh. Yeah, but Ggl, LM five, right?

We, we have this, new model. It’s, it’s like a large size, and you have to figure out how to both scale up and scale out, right? Because you have to find the right representation that you care about. Everyone does this differently. Let’s be very clear. Everyone figures this out in their own path.

Nader: I feel like a lot of AI or ML even is like, is like this. I think people think, you know, I, I was, there was some tweet a few months ago that was like, why hasn’t fine tuning as a service taken off? You know, that might be me. It might have been you. Yeah. But people want it to be such an easy recipe to follow.

But even like if you look at an ML model and specific

Kyle: to you Yeah,

Nader: yeah.

Kyle: And the [00:32:00] model,

Nader: the situation, and there’s just so much tinkering, right? Like when you see a model that has however many experts in the ME model, it’s like, why that many experts? I don’t, they, you know, they tried a bunch of things and that one seemed to do better.

I think when it comes to how you’re serving inference, you know, you have a bunch of decisions to make and there you can always argue that you can take something and make it more optimal. But I think it’s this internal calibration and appetite for continued calibration.

Vibhu: Yeah. And that doesn’t mean like, you know, people aren’t taking a shot at this, like tinker from thinking machines, you know?

Yeah. RL as a service. Yeah, totally. It’s, it also gets even harder when you try to do big model training, right? We’re not the best at training Moes, uh, when they’re pre-trained. Like we saw this with LAMA three, right? They’re trained in such a sparse way that meta knows there’s gonna be a bunch of inference done on these, right?

They’ll open source it, but it’s very trained for what meta infrastructure wants, right? They wanna, they wanna inference it a lot. Now the question to basically think about is, okay, say you wanna serve a chat application, a coding copilot, right? You’re doing a layer of rl, you’re serving a model for X amount of people.

Is it a chat model, a coding model? Dynamo, you know, back to that,

Kyle: it’s [00:33:00] like, yeah, sorry. So you we, we sort of like jumped off of, you know, jumped, uh, on that topic. Everyone has like, their own, own journey.

Cost Quality Latency Tradeoffs

Kyle: And I, I like to think of it as defined by like, what is the model you need? What is the accuracy you need?

Actually I talked to NA about this earlier. There’s three axes you care about. What is the quality that you’re able to produce? So like, are you accurate enough or can you complete the task with enough, performance, high enough performance. Yeah, yeah. Uh, there’s cost. Can you serve the model or serve your workflow?

Because it’s not just the model anymore, it’s the workflow. It’s the multi turn with an agent cheaply enough. And then can you serve it fast enough? And we’re seeing all three of these, like, play out, like we saw, we saw new models from OpenAI that you know, are faster. You have like these new fast versions of models.

You can change the amount of thinking to change the amount of quality, right? Produce more tokens, but at a higher cost in a, in a higher latency. And really like when you start this journey of like trying to figure out how you wanna host a model, you, you, you think about three things. What is the model I need to serve?

How many times do I need to call it? What is the input sequence link was [00:34:00] the, what does the workflow look like on top of it? What is the SLA, what is the latency SLA that I need to achieve? Because there’s usually some, this is usually like a constant, you, you know, the SLA that you need to hit and then like you try and find the lowest cost version that hits all of these constraints.

Usually, you know, you, you start with those things and you say you, you kind of do like a bit of experimentation across some common configurations. You change the tensor parallel size, which is a form of parallelism

Vibhu: I take, it goes even deeper first. Gotta think what model.

Kyle: Yes, course,

of

Kyle: course. It’s like, it’s like a multi-step design process because as you said, you can, you can choose a smaller model and then do more test time scaling and it’ll equate the quality of a larger model because you’re doing the test time scaling or you’re adding a harness or something.

So yes, it, it goes way deeper than that. But from the performance perspective, like once you get to the model you need, you need to host, you look at that and you say, Hey. I have this model, I need to serve it at the speed. What is the right configuration for that?

Nader: You guys see the recent, uh, there was a paper I just saw like a few days ago that, uh, if you run [00:35:00] the same prompt twice, you’re getting like double Just try it

again.

Nader: Yeah, exactly.

Vibhu: And you get a lot. Yeah. But the, the key thing there is you give the context of the failed try, right? Yeah. So it takes a shot. And this has been like, you know, basic guidance for quite a while. Just try again. ‘cause you know, trying, just try again. Did you try again? All advice

Nader: in life.

Vibhu: Just, it’s a paper from Google, if I’m not mistaken, right?

Yeah,

Vibhu: yeah. I think it, it’s like a seven bas little short paper. Yeah. Yeah. The title’s very cute. And it’s just like, yeah, just try again. Give it ask context,

Kyle: multi-shot. You just like, say like, hey, like, you know, like take, take a little bit more, take a little bit more information, try and fail. Fail.

Vibhu: And that basic concept has gone pretty deep.

There’s like, um, self distillation, rl where you, you do self distillation, you do rl and you have past failure and you know, that gives some signal so people take, try it again. Not strong enough.

swyx: Uh, for, for listeners, uh, who listen to here, uh, vivo actually, and I, and we run a second YouTube channel for our paper club where, oh, that’s awesome.

Vivo just covered this. Yeah. Awesome. Self desolation and all that’s, that’s why he, to speed [00:36:00] on it.

Nader: I’ll to check it out.

swyx: Yeah. It, it’s just a good practice, like everyone needs, like a paper club where like you just read papers together and the social pressure just kind of forces you to just,

Nader: we, we,

there’s

Nader: like a big inference.

Kyle: Reading

Nader: group at a video. I feel so bad every time. I I, he put it on like, on our, he shared it.

swyx: One, one of

Nader: your guys,

swyx: uh, is, is big in that, I forget es han Yeah, yeah,

Kyle: es Han’s on my team. Actually. Funny. There’s a, there’s a, there’s a employee transfer between us. Han worked for Nater at Brev, and now he, he’s on my team.

He was

Nader: our head of ai. And then, yeah, once we got in, and

swyx: because I’m always looking for like, okay, can, can I start at another podcast that only does that thing? Yeah. And, uh, Esan was like, I was trying to like nudge Esan into like, is there something here? I mean, I don’t think there’s, there’s new infant techniques every day.

So it’s like, it’s like

Kyle: you would, you would actually be surprised, um, the amount of blog posts you see. And if

swyx: there’s a period where it was like, Medusa hydra, what Eagle, like, you

Kyle: know, now we have new forms of decode, uh, we have new forms of specula, of decoding or new,

swyx: what,

Kyle: what are you

Vibhu: excited? And it’s exciting when you guys put out something like Tron.

‘cause I remember the paper on this Tron three, [00:37:00] uh, the amount of like post train, the on tokens that the GPU rich can just train on. And it, it was a hybrid state space model, right? Yeah.

Kyle: It’s co-designed for the hardware.

Vibhu: Yeah, go design for the hardware. And one of the things was always, you know, the state space models don’t scale as well when you do a conversion or whatever the performance.

And you guys are like, no, just keep draining. And Nitron shows a lot of that. Yeah.

Nader: Also, something cool about Nitron it was released in layers, if you will, very similar to Dynamo. It’s, it’s, it’s essentially it was released as you can, the pre-training, post-training data sets are released. Yeah. The recipes on how to do it are released.

The model itself is released. It’s full model. You just benefit from us turning on the GPUs. But there are companies like, uh, ServiceNow took the dataset and they trained their own model and we were super excited and like, you know, celebrated that work.

Zoom

Vibhu: different. Zoom is, zoom is CGI, I think, uh, you know, also just to add like a lot of models don’t put out based models and if there’s that, why is fine tuning not taken off?

You know, you can do your own training. Yeah,

Kyle: sure.

Vibhu: You guys put out based model, I think you put out everything.

Nader: I believe I know [00:38:00]

swyx: about base. Basically

Vibhu: without base

swyx: basic can be cancelable.

Vibhu: Yeah. Base can be cancelable.

swyx: Yeah.

Vibhu: Safety training.

swyx: Did we get a full picture of dymo? I, I don’t know if we, what,

Nader: what I’d love is you, you mentioned the three axes like break it down of like, you know, what’s prefilled decode and like what are the optimizations that we can get with Dynamo?

Kyle: Yeah. That, that’s, that’s, that’s a great point. So to summarize on that three axis problem, right, there are three things that determine whether or not something can be done with inference, cost, quality, latency, right? Dynamo is supposed to be there to provide you like the runtime that allows you to pull levers to, you know, mix it up and move around the parade of frontier or the preto surface that determines is this actually possible with inference And AI today

Nader: gives you the knobs.

Kyle: Yeah, exactly. It gives you the knobs.

Disaggregation Prefill vs Decode

Kyle: Uh, and one thing that like we, we use a lot in contemporary inference and is, you know, starting to like pick up from, you know, in, in general knowledge is this co concept of disaggregation. So historically. Models would be hosted with a single inference engine. And that inference engine [00:39:00] would ping pong between two phases.

There’s prefill where you’re reading the sequence generating KV cache, which is basically just a set of vectors that represent the sequence. And then using that KV cache to generate new tokens, which is called Decode. And some brilliant researchers across multiple different papers essentially made the realization that if you separate these two phases, you actually gain some benefits.

Those benefits are basically a you don’t have to worry about step synchronous scheduling. So the way that an inference engine works is you do one step and then you finish it, and then you schedule, you start scheduling the next step there. It’s not like fully asynchronous. And the problem with that is you would have, uh, essentially pre-fill and decode are, are actually very different in terms of both their resource requirements and their sometimes their runtime.

So you would have like prefill that would like block decode steps because you, you’d still be pre-filing and you couldn’t schedule because you know the step has to end. So you remove that scheduling issue and then you also allow you, or you yourself, to like [00:40:00] split the work into two different ki types of pools.

So pre-fill typically, and, and this changes as, as model architecture changes. Pre-fill is, right now, compute bound most of the time with the sequence is sufficiently long. It’s compute bound. On the decode side because you’re doing a full Passover, all the weights and the entire sequence, every time you do a decode step and you’re, you don’t have the quadratic computation of KV cache, it’s usually memory bound because you’re retrieving a linear amount of memory and you’re doing a linear amount of compute as opposed to prefill where you retrieve a linear amount of memory and then use a quadratic.

You know,

Nader: it’s funny, someone exo Labs did a really cool demo where for the DGX Spark, which has a lot more compute, you can do the pre the compute hungry prefill on a DG X spark and then do the decode on a, on a Mac. Yeah. And so

Vibhu: that’s faster.

Nader: Yeah. Yeah.

Kyle: So you could, you can do that. You can do machine strat stratification.

Nader: Yeah.

Kyle: And like with our future generation generations of hardware, we actually announced, like with Reuben, this [00:41:00] new accelerator that is prefilled specific. It’s called Reuben, CPX. So

Kubernetes Scaling with Grove

Nader: I have a question when you do the scale out. Yeah. Is scaling out easier with Dynamo? Because when you need a new node, you can dedicate it to either the Prefill or, uh, decode.

Kyle: Yeah. So Dynamo actually has like a, a Kubernetes component in it called Grove that allows you to, to do this like crazy scaling specialization. It has like this hot, it’s a representation that, I don’t wanna go too deep into Kubernetes here, but there was a previous way that you would like launch multi-node work.

Uh, it’s called Leader Worker Set. It’s in the Kubernetes standard, and Leader worker set is great. It served a lot of people super well for a long period of time. But one of the things that it’s struggles with is representing a set of cases where you have a multi-node replica that has a pair, right?

You know, prefill and decode, or it’s not paired, but it has like a second stage that has a ratio that changes over time. And prefill and decode are like two different things as your workload changes, right? The amount of prefill you’ll need to do may change. [00:42:00] The amount of decode that you, you’ll need to do might change, right?

Like, let’s say you start getting like insanely long queries, right? That probably means that your prefill scales like harder because you’re hitting these, this quadratic scaling growth.

swyx: Yeah.

And then for listeners, like prefill will be long input. Decode would be long output, for example, right?

Kyle: Yeah. So like decode, decode scale. I mean, decode is funny because the amount of tokens that you produce scales with the output length, but the amount of work that you do per step scales with the amount of tokens in the context.

swyx: Yes.

Kyle: So both scales with the input and the output.

swyx: That’s true.

Kyle: But on the pre-fold view code side, like if.

Suddenly, like the amount of work you’re doing on the decode side stays about the same or like scales a little bit, and then the prefilled side like jumps up a lot. You actually don’t want that ratio to be the same. You want it to change over time. So Dynamo has a set of components that A, tell you how to scale.

It tells you how many prefilled workers and decoded workers you, it thinks you should have, and also provides a scheduling API for Kubernetes that allows you to actually represent and affect this scheduling on, on, on your actual [00:43:00] hardware, on your compute infrastructure.

Nader: Not gonna lie. I feel a little embarrassed for being proud of my SVG function earlier.

swyx: No, it

Nader: was

really

Kyle: cute. I, I

swyx: like

Nader: it’s all,

swyx: it’s all engineering. It’s all engineering. Um, that’s where I’m

Kyle: technical.

swyx: One thing I’m, I’m kind of just curious about with all with you see at a systems level, everything going on here. Mm-hmm. And we, you know, we’re scaling it up in, in multi, in distributed systems.

Context Length and Co Design

swyx: Um, I think one thing that’s like kind of, of the moment right now is people are asking, is there any SOL sort of upper bounds. In terms of like, let’s call, just call it context length for one for of a better word, but you can break it down however you like.

Nader: Yeah.

swyx: I just think like, well, yeah, I mean, like clearly you can engage in hybrid architectures and throw in some state space models in there.

All, all you want, but it looks, still looks very attention heavy.

Kyle: Yes. Uh, yeah. Long context is attention heavy. I mean, we have these hybrid models, um,

swyx: to take and most, most models like cap out at a million contexts and that’s it. Yeah. Like for the last two years has been it.

Kyle: Yeah. The model hardware context co-design thing that we’re seeing these days is actually super [00:44:00] interesting.

It’s like my, my passion, like my secret side passion. We see models like Kimmy or G-P-T-O-S-S. I’m use these because I, I know specific things about these models. So Kimmy two comes out, right? And it’s an interesting model. It’s like, like a deep seek style architecture is MLA. It’s basically deep seek, scaled like a little bit differently, um, and obviously trained differently as well.

But they, they talked about, why they made the design choices for context. Kimmy has more experts, but fewer attention heads, and I believe a slightly smaller attention, uh, like dimension. But I need to remember, I need to check that. Uh, it doesn’t matter. But they discussed this actually at length in a blog post on ji, which is like our pu which is like credit pu

swyx: Yeah.

Kyle: Um, in, in China. Chinese red.

swyx: Yeah.

Kyle: It’s, yeah. So it, it’s, it’s actually an incredible blog post. Uh, like all the mls people in, in, in that, I’ve seen that on GPU are like very brilliant, but they, they talk about like the creators of Kimi K two [00:45:00] actually like, talked about it on, on, on there in the blog post.

And they say, we, we actually did an experiment, right? Attention scales with the number of heads, obviously. Like if you have 64 heads versus 32 heads, you do half the work of attention. You still scale quadratic, but you do half the work. And they made a, a very specific like. Sort of barter in their system, in their architecture, they basically said, Hey, what if we gave it more experts, so we’re gonna use more memory capacity.

But we keep the amount of activated experts the same. We increase the expert sparsity, so we have fewer experts act. The ratio to of experts activated to number of experts is smaller, and we decrease the number of attention heads.

Vibhu: And kind of for context, what the, what we had been seeing was you make models sparser instead.

So no one was really touching heads. You’re just having, uh,

Kyle: well, they, they did, they implicitly made it sparser.

Vibhu: Yeah, yeah. For, for Kimmy. They did,

Kyle: yes.

Vibhu: They also made it sparser. But basically what we were seeing was people were at the level of, okay, there’s a sparsity ratio. You want more total parameters, less active, and that’s sparsity.[00:46:00]

But what you see from papers, like, the labs like moonshot deep seek, they go to the level of, okay, outside of just number of experts, you can also change how many attention heads and less attention layers. More attention. Layers. Layers, yeah. Yes, yes. So, and that’s all basically coming back to, just tied together is like hardware model, co-design, which is

Kyle: hardware model, co model, context, co-design.

Vibhu: Yeah.

Kyle: Right. Like if you were training a, a model that was like. Really, really short context, uh, or like really is good at super short context tasks. You may like design it in a way such that like you don’t care about attention scaling because it hasn’t hit that, like the turning point where like the quadratic curve takes over.

Nader: How do you consider attention or context as a separate part of the co-design? Like I would imagine hardware or just how I would’ve thought of it is like hardware model. Co-design would be hardware model context co-design

Kyle: because the harness and the context that is produced by the harness is a part of the model.

Once it’s trained in,

Vibhu: like even though towards the end you’ll do long context, you’re not changing architecture through I see. Training. Yeah.

Kyle: I mean you can try.

swyx: You’re saying [00:47:00] everyone’s training the harness into the model.

Kyle: I would say to some degree, or

swyx: there’s co-design for harness. I know there’s a small amount, but I feel like not everyone has like gone full send on this.

Kyle: I think, I think I think it’s important to internalize the harness that you think the model will be running. Running into the model.

swyx: Yeah. Interesting. Okay. Bash is like the universal harness,

Kyle: right? Like I’ll, I’ll give. An example here, right? I mean, or just like a, like a, it’s easy proof, right? If you can train against a harness and you’re using that harness for everything, wouldn’t you just train with the harness to ensure that you get the best possible quality out of,

swyx: Well, the, uh, I, I can provide a counter argument.

Yeah, sure. Which is what you wanna provide a generally useful model for other people to plug into their harnesses, right? So if you

Kyle: Yeah. Harnesses can be open, open source, right?

swyx: Yeah. So I mean, that’s, that’s effectively what’s happening with Codex.

Kyle: Yeah.

swyx: And, but like you may want like a different search tool and then you may have to name it differently or,

Nader: I don’t know how much people have pushed on this, but can you.

Train a model, would it be, have you have people compared training a model for the for the harness versus [00:48:00] like post training for

swyx: I think it’s the same thing. It’s the same thing. It’s okay. Just extra post training. I

Nader: see.

swyx: And so, I mean, cognition does this course, it does this where you, you just have to like, if your tool is slightly different, um, either force your tool to be like the tool that they train for.

Hmm. Or undo their training for their tool and then Oh, that’s re retrain. Yeah. It’s, it’s really annoying and like,

Kyle: I would hope that eventually we hit like a certain level of generality with respect to training new

swyx: tools. This is not a GI like, it’s, this is a really stupid like. Learn my tool b***h.

Like, I don’t know if, I don’t know if I can say that, but like, you know, um, I think what my point kind of is, is that there’s, like, I look at slopes of the scaling laws and like, this slope is not working, man. We, we are at a million token context, okay, maybe next year, 2 million, we’re not going to a hundred trillion, you know, like this, this, oh, there’s so many interesting ways to get this Doesn’t work.

Just doesn’t work.

Nader: What’s kind of funny is whenever there, I, I feel like we always want to see a trend that we can predict, but every time something’s come, it’s been like a leapfrog. So I, I imagine I, I don’t know how we go from one to two, but I imagine what, what’s likely to happen is [00:49:00] we break through that from some new

Kyle: Yeah.

There’s actually, there’s an interesting formalization of this. There, there’s an essay. It’s a pretty interesting essay by Leopold Ashton Brener called Situational Awareness.

swyx: Okay? Yes.

Kyle: He introduces a concept awareness called an un hobbler, right? So he, you know, Leopold in this essay details, Hey, I want to get.

You know, like, I wanna get to this point in intelligence and I think that it is four orders of magnitude worth of like compute and data and training away. And you know, he says, oh yeah, I think data centers can scale up by about this much. I think that you can do, scale up the data and some other things by this much.

But one of the things that like makes the rest of that order of magnitude growth, PO possibilities is un hobbler, like these scientific discoveries that are discovered during. You know, model architecture, search or training that really, really, really impact how, how you are able to scale. Like a, a good example of this might be that like we see like a mo a lot of models that are, [00:50:00] and this is probably a very tiny on hobbler.

But is important for the performance perspective. We see a lot of models that are like trained with multi token prediction natively in during pre-training.

And per deep seek in their paper they say, Hey, decided this actually helped us in ensure sta more stable convergence. But they’re like, un Hobbs that are like that.

And then they’re like, rather large on hobbler. Right. Like architecturally, a lot of our models, like we had different types of attention. And one of the problems with attention is like, you have a lot of kv, but people found like different forms of attention, like group query attention and, uh, like MLA in deep seek multi-head latent attention that like decrease the burden that KV has on the model, which allows you to grow like longer in context.

swyx: Yeah. And that, that was very drastic for deeps seek.

Kyle: Yeah. This was like, yeah, it for context like the, the total, I think the total context length of deeps seek is 128,000 tokens or might be 256,000 with rope extension. That entire context, I think it’s 128,000 fits into eight gigabytes. Previously context, like I think the, the llama four or five B context [00:51:00] of a similar size was like 40 or 80 gigabytes in the same precision.

swyx: Yeah.

Kyle: Um, so like those in Hobbler like really decrease the stuff of that size. And I wouldn’t be surprised if we do see the ability to like, break through to like 10 million, 20 million, a hundred million context through the an un hobbler showing up. I

swyx: see.

Kyle: And it’s just science.

swyx: So more deep learning algorithms is what

Kyle: I’m hearing.

Yeah. More deep learning algorithms. Um,

swyx: yeah,

Kyle: I, I could, I could actually playing pickup

swyx: and he has

Kyle: room to, I I could actually give you an, an example like of like a, a theory, not a theory theory, but something theoretical and a hobar

Nader: that you’re excited about or,

Kyle: well, and, and a hobar that, I mean, I haven’t seen, so it could be a tar pit and it could not, just, not work.

But, uh, I, I would be really excited to see a model that does prefill and decode differently. So a model that does, uh, prefill like locally, like document wise, prefill, like it doesn’t in chunks, and then you do decode globally across like the entire sequence because it, logically to me it doesn’t seem like you would necessarily need to [00:52:00] have KV b associative between documents that have like, no, no mutual association.

But that like places a lot of burden on prefilled to like, or sorry, on, on decode and pure attention within the decode phase to like make those connections since the KV is like static at that point. And you see other techniques that are interesting like this too. But if, if you’re able to do that, like.

If Prefill becomes local and decode is, is still global, you solve that prefilled quadratic scaling problem because you have a bunch of like small chunks that you prefill independently.

swyx: Okay. All right. Well, let’s, uh, wait and see, but I, I think it’ll be pretty exciting.

Kyle: Fingers crossed.

swyx: Yeah, fingers crossed.

Yeah. Yeah.

Vibhu: I’m excited for prefilled decode on separate hardware. So like yeah. CR acquisition, right. Can we decode on the gr Can we get super fast?

Kyle: I don’t think I’m allowed to comment on this.

swyx: Mark is gonna shoot arrows at us.

Nader: Uh, he’s got a blow dark, he’s in the room, just

Kyle: like,

Nader: like go to sleep.

Yeah. Yeah.

swyx: But

Nader: I’m, I’m super excited to see the team come in and like, you know, I’ve gotten the, the pleasure of working with some of the, the GR people coming in. So, you know, yeah, I,

swyx: I know Sonny, [00:53:00] we’ve had him, uh, at the same

Kyle: conference that

swyx: you are at.

Nader: Yeah.

swyx: Um, and, uh, I, I think you’re, you guys are gonna be doing some sessions at G tc.

I don’t know if you wanna, this is a good place to plug them.

Kyle: Yeah, yeah, yeah. So, I can’t speak to any LPU related sessions at G tc. I have no idea about that. Oh, no, that was,

swyx: no. Yours

Kyle: on the, on the GR side. Yeah. I use the associative NVIDIA U Yeah. Um, on the, on the Nvidia Dynamo side, we’re, we’re giving, there are a large number of sessions.

For those that aren’t aware, you can actually search. All of these sessions for GTC online, just go to the GTC website. I don’t know what the URL is, but go there. Google it. Yeah. Uh, and you can just look up Dynamo and you’ll get all the sessions. There’re about 20. There are a couple that are hosted by the Dynamo team.

There are a couple that are hosted by people that use Dynamo that wanna show off the results they’ve been able to get. But there are two that I’m really excited about. Uh, one is just the General Dynamo tutorial, and this is the, I’m going out with Harry, who’s our lead product manager for Dynamo.

And we’re sort of talking about like how to use Dynamo to get better performance and also like where we see Dynamo going in the future. And [00:54:00] then there’s another session that I’m doing with one of our agents teams at Nvidia to talk about sort of the future of agents in production inference. Yeah. So we’re talking about, there’s like this new horizon with respect to agents because we have these harnesses that actually impart structure on upon calls.

Like if you, if you compare like, the past and the, and the present with respect to like how LM calls work. Like in the early days when they were chatbots, like every call was like very different. There was basically no structure. You could assume that like people, you, if it was conversational, there might be like some implicit structure because you have, you know, a multi-term conversation.

But agency have this, this harness that, like abides by rules, right? So it imparts direct structure onto the context. And you see this, there was an interesting Twitter post about how Claude code like structures, its context so that you get as many cts as possible.

And I think it was by one of the, the PMs for Claude code.

And he, he wrote about it. And that type of structure that the harness can impart actually like goes hand in [00:55:00] hand with the. Inference co-design. So I’m doing a talk, I, I don’t know the session name or the session number, but I’m, I’m doing a talk, uh, you can look at me up by name on, on the GTC website, on how we accelerate agents and where we see specific optimizations for agents going in Dynamo and in inference in general.

swyx: Yeah. I think there’s only 1:00 PM for cloud code and it’s wo the rest. There’s, there’s Devrel, there’s Boris. Maybe it was maybe Devrel. Yeah, exactly. I mean, let’s go into agents. I think this was like the last part of the, the, the discussion we planned. Yeah. How have we not talked about agents also with you guys?

Well, we scheduled, it was like, I was like, okay, you know, like, let’s have like cohesive sections or,

Vibhu: I mean, there’s the big news, right? The NVIDIA’s a huge. Like deployment of Codex. Yeah, video

swyx: uses everything. I mean, we use this cursor and we uses code,

Vibhu: but that’s, that’s a pretty big deployment, right?

Like, that’s tens of thousands of people.

Nader: Totally. Yeah.

Vibhu: We’re super What? That’s,

Nader: yeah. I, it goes back to the mosh pit of emails we kind of mentioned earlier, or just the like, um, how fluid the org feels. So when there’s new technology, people will just email it out and everyone will try it.

[00:56:00] And if it, if it’s making people’s lives easier, it’ll spread like wildfire.

Kyle: A lot of times Jensen will get it and it’ll be like, let’s make this work. Yeah. Across the company. Let’s make this work right now,

Nader: honestly, uh, if I was a startup, I feel like a cool hack. If you have something that’s going to save an Nvidia time they’ll spread it to a couple and the same thing.

Right? It’ll just spread like wildfire. Okay.

Vibhu: Careful before your email blows up from startups. Well,

Nader: You gotta know the person. Right? But no, I, um, I, yeah, so I mean, we, I love using Codex. It’s been a ton of fun. Yeah. Uh, I’ve been using it personally. I’ve been using it at work. It’s been, um.

Yeah, I dunno. It’s been great to see the rollout, something really funny. Uh, on the data we got, uh, codex and cloud code access. I found this person, uh, his name’s Carlos at the company. He wrote an Outlook, CLI.

Kyle: Oh yeah.

Nader: And, uh, just the CLI for email. And this was, I’ve

Kyle: been using that,

Nader: yeah, maybe like four or five weeks ago.

And, uh, the site, so once I got like Codex access I. Installed the CLI, it had a skill and I just asked it to go through all of my emails, which it’s very messy. So if I don’t respond to your email, I’m really sorry. But I asked it to gimme a summary, highlight any [00:57:00] escalations that I should look at, put any thread that it thinks I should respond to in a folder, and then archive everything.

And it did. So if I missed your email, it’s because it didn’t get,

swyx: so I should put a prompt injection in my V to Yeah, yeah. What you should do is just FaceTime. Yeah. Um, my, yeah, my SLA is highest on FaceTime,

Nader: but that was, it was magic. And so I, I sent it in a big email thread to like 500 people. A bunch of folks tried it out.

I started like FaceTiming whoever I could at the company to get them set up with this.

swyx: Yeah. Um, that specific example mm-hmm. You guys deal with like some pretty. Sensitive emails.

Nader: Yeah.

swyx: Is there a security review with this?

Security Meets Agents

swyx: ‘cause like one guy made, made it for himself, but like it’s not meant for all the

Nader: security team and Nvidia is incredible.

Like, shout out to them. They’re, they’re, they’re trying to, we have a, we have an amazing security team ‘cause they’re progressive and they know that this is

Kyle: really important technology and you have to bring it in. If you think about like, if you work at a big company, your laptop’s usually very locked down if

Nader: you can only access certain things.

Nvidia engineers have those restrictions aren’t there. So you’re expected to understand the risks when you try things out. And so. Very quickly, you know, made sure to [00:58:00] chime in security on what we were doing.

Agent Permissions Model

Nader: There’s actually a lot that we’ve been thinking about, especially with open claw, right? Like there’s, you know, agents can do three things.

Yeah. A agents can do three things. They can access your files, they can access the internet, and then now they can write custom code, uh, and execute it. And you literally only let an agent do two of those three things. If you can access your files and you can write custom code, you don’t want internet access because that’s one to see full vulnerability, right?

If you have access to internet and your file system, you should know the full scope of what that agent’s capable of doing. Otherwise, malware can get injected or something that can happen. And so that’s a lot of what we’ve been thinking about is like, you know, how do we both enable this because it’s clearly the future.

But then also, you know, what, what are these enforcement points that we can start to like protect?

swyx: And is there any directive of like, Hey, we have a company account or a company agreement with open ai, we use open AI models here, or like choose whatever.

Nader: No, no. So, so I would never put any company data in a model that’s not either, that we don’t even, it has to most security.

Yeah. Yeah. I like how,

swyx: how that goes. Uh, you know, obviously you could run your own [00:59:00] models. You Nemo and, and we, right, we, we as an, we have an internal cluster, so, you know, of course in random,

Kyle: uh, yeah.

swyx: Yeah.

Nader: I think we’re dynamo’s first customers. Let’s go

Build Nvidia Inference Gateway

Kyle: actually, uh, there’s a funny story about like how I got the experience that informed what we needed for Dynamo at one point.

There’s a website called build done n video.com and also for us infra dun n video com. That is allows people to try models. It gives an a p service. You can call the model with like a rest, API, and you know, you get a response. I ran the model side for that and it was at one point the largest inference deployment and still may actually be the largest inference deployment in video.

I’ve, I’ve since like, handed it off to some people and they’re doing a wonderful by way. This is a extremely

Nader: underknown or less known resource. Vil diamond v.com. You can get any of these open source models. And it’s rate limited, but it’s free. So it’s perfect for hackers to,

Kyle: and, and the SLA on getting models day zero models up is like a day.

Yeah.

Kyle: Like they’re, they’re incredibly good at like figuring out the right way to host the model to [01:00:00] get it up there as soon as it comes out.

swyx: You ran this?

Kyle: Yeah, I ran, I ran it a long time ago. It was originally called Nvidia AI Playground, then it was called AI Foundational insert. Yeah. And then it was called Build Nvidia call.

And I, I ran the model side of it. So there were, there was a large multi-organizational team. I ran how, which models should we host? How should we host them and like what’s the proportion of them? And then of course there was like an SRE team that like made sure that things ran well and scaled the models as well.

But I ran like, you know, model, how do we get the model to silicon? And then, which model also worked with our product team Determine like which models were important a very long time ago.

Yeah. Yeah. There’s also like a middle ground in between there, right? This is like for the hacker. Try anything.

There’s the Brev console, then there’s Dynamo, there was also nims, right?

Kyle: Yes.

I remember it had its little moment, like a year or two ago. Is it still?

Nader: Yeah. NIM is, uh, you know, inference, uh, oil. I, I think it like for something is it is a log or acronym. Yeah. It [01:01:00] just, just a name. But, um, yeah, NIM is, uh, how enterprises can take our uh, any of the, any of this technology and run it with support and all of that.

And so that includes Daniel Mo. That includes, I don’t know all of our other optimizations that are packers up for Enterprise. Yep.

swyx: Anyway, so, so you, you got a bunch of experience start running the sort of internal inference gateway playgrounds.

Kyle: Yeah, I got And Bill also built how build NVIDIA’s first internal like vs.

Code thing. We call it MB code.

swyx: That’s what I, uh, extension.

Kyle: Yeah, it was, it was a V first,

like the fork vs code.

swyx: We jokes absolutely not. It just a while back they like, we should have a fourth vs. Code hackathon where you, that’s four. It’s the best four V vs code. We,

we were, we were doing a hack how make a billion dollars, someone from VS code was there and he was like somewhat down to get involved and I was like,

swyx: oh, you should do that.

That’s all. Then the cool thing became four chrome hackathon

Chrome,

swyx: And no, no, no IDs or not cooling.

Nader: I saw, what’s it called?

Hackathons And Autonomy Dreams

Nader: I was talking to Joseph, uh, from Robo Flow and uh, they’re partnering crime. We were talking about how with the new Alpha Mayo model, so Nvidia just [01:02:00] released an open source. Uh, the, the Mercedes cars that you saw drag, she on Frazey?

swyx: Yeah.

Nader: Released. Will you open source, a autonomous driving model? Uh, I already, yeah, so we were thinking like, could we hackathon a driverless car? Like I have my old car. Let’s just try it.

swyx: We’ll take it,

Nader: take it to like, click train with a treasure eye, like in the middle of the day. Just like, just see, let everyone, like how many, how many cameras do we need?

Right? Like, 1, 2,

swyx: 3, 4. They don’t. Five, six.

Nader: I don’t know. I, yeah. But, um, I think we’re gonna try, you just do it with us.

swyx: We can see, we could even

Nader: have a race. It’s like the first person to automate their

swyx: driving. Let me over a weekend. We do have an autonomy track at Will’s fair. Uh, WiMo was there like Yeah.

Nvidia did send people that for Goot. Not because he didn’t have the driving thing yet.

Nader: Yeah.

swyx: Yeah. It’s, that’s cool.

Yeah. I think comma, comma also has a version of this comma have open source driving. They’ve, they’ve done a fun hackathon on

swyx: music and he and I also, ‘cause I, I really, what I really want is a Tesla with Tesla level self-driving.

Yeah.

swyx: But as a smart car, like a two seater. That’s the basic CPA wheelchair with a [01:03:00] roof

and only thing they make them, but the demand has d they, no, they realize this probably five years. Yeah. Really?

swyx: Yeah.

They were d manufacturer.

Kyle: I thought it is one of those things, we’ll, where we’ll see someone buy the brand and it’ll be revived.

swyx: I, I would buy it like I

Kyle: probably. Someone hears this go by

swyx: your car. Yeah. Yeah. That’s crazy. Nobody Mercedes, because they, they’re like, I think 10 Mercedes, Mercedes, uh, I in Mercedes used

to make them, I don’t know. I feel like they own the brand and you out

swyx: that’s your dream might come true enough. Okay.

We we’re time notify and, and I was like, every time I, I try to park in San Francisco, I I have to buy a smart car because like 20% of the parking lots in San Francisco only fit smart cars.

Nader: Yeah. So, Hey, really?

swyx: That’s where, I mean, it’s mall

Nader: even it was late here trying to, this comes from someone that like, basically does

Kyle: not drive.

Nader: That’s where the, the Vepa was a life hack. Yeah, exactly. Yeah. You know what happened to the Vespa? Um, I used to have [01:04:00] this yellow Vespa, uh, I left it outside the hacker house when we moved out. It trend. Um, it’s just, it was always there. And then like a month ago. It’s not there anymore. I’ve been meeting today.

I don’t dunno. You could, it’s actually tv. You forgot about it.

swyx: Yeah.

Nader: And left.

swyx: Yeah. Yeah. No, this, it’s probably hazard. And speaking of hackathons, I also wanted say, give a big shout out to the world. Shortest hackathon. Let’s go. Uh, you did twice. You gonna watch a

Nader: handful of times? Yeah. There’s gonna be one at G tc.

Oh, we’re doing pretty much we have a bunch of challenges that No, we haven’t released. And you get to bring your agent to come and attempt to, uh, go through those

Kyle: challeng again. It’s like a zero, the zero minute hackathon idea, which you just, you just bring your, I I approached eight, nine along a long time ago.

You just bring your agent and then you press the go button. You’re not allowed to code. It’s just the Asian doing bond.

It’s a good hidden email, right?

Kyle: Yeah.

Do you make a jar? You make

Kyle: I there something I would love to see from cognition or someone else be like, come bring your agent. Drop it in

because you don’t, you don’t know you like supervisor.

Well let be [01:05:00] a, you know, operate a browser, order a pizza. We’ll just see like that snake it, you know,

swyx: and

Kyle: you don’t know what the

swyx: task

Kyle: is. Yeah. You dunno what the task is like, or just like, you don’t even know what the judging categories are and then you give it the judging categories. Like, try as much as possible.

It’s great though. It turns into like, yeah, so let’s build something on dining party. It’s a great business. See,

Kyle: anyway, funny story.

Agent UX And CLI Everywhere

Kyle: Actually, we have a couple of people at Nvidia, we’ve been working with security to like bring agents really close to compute. So we now have like stuff where we can like tell Dynamo, like go run some experience with Dynamo, like on, X cluster and just like try it right now, like queue up once you get queued, like, send this request load and we’ve actually been able to like, just like, you know, like one shot problems like.

We used to have this problem where you know, with Dynamo you have to like find the right configurations and we, sort of do it automatically for some parts of it, but you have to like a good initial configuration that you want to use. And we’ve just had like an agent just completely one shot that it goes, it gets the compute, it like runs a couple experiments.

It’s like [01:06:00] this is the best, this is this, these are part of the ER frontier. Go run this. And then we just like give that to people and it’s like faster than anything that they have.

Nader: Agent UX and agent marketing are super important. There’s stuff that we’ve been thinking a lot about. Um, Alec is like redoing the entire Brev CLI, um, so that you can fetch all the different compute types that are available.

I don’t know, it’s gonna be really soon, but then you can, you can just browse what GPUs are available and then provision one say to it right there. And you can pipe all the commands. But I think it goes back to like the Alex CLI, like if you, coding agents. It’s kind of funny. I feel like coding agents have been so much more effective than general purpose agents.

And I think a large part of that is it just has access to the terminal, like you said, and that means it has access to everything that you’ve installed into your terminal. It can run. So, you know, it would write code and, and it can compile the code and if there are errors, it can fix it, it can run your suite of tests because that’s all just in your terminal.

And so that, you know, then for the idea, what come me really excited about the CLI, we’re now just turning through building CLI for the entire, like for the entire business. We Slack, building Slack, also. Workday, C-L-I-S-A Go. I, I’ve also done that for myself first. Really? Yeah. Yeah. Um, we’re gonna, we’re gonna [01:07:00] open source all of this.

And like yeah, all the, the I they’re just they’re the C yeah. CLI for the business applications. We would love for someone to run with this and like build like, I don’t know, like open CLI foundation in or something. Yeah. We, I Nvidia would love to support, uh, anyone that’s doing this.

Like e every Devrel tool should really have good CLI support at this point.

Yeah. Like at one point it was, you want your docs to be. Like accessible by an LM, right? You want LM Good dog. No, every, everything needs some CLI.

Nader: Yeah. It’s kind of funny, right? Like we, like computing began with a terminal with a shell, but we said that it’s not empathetic to, uh, humans. So we built these nice user interfaces and then now we have LMS navigating our user interfaces.

And ironically, we’re not empathetic to the machine anymore.

swyx: Yeah.

Nader: Yeah. Just give the, the LLM access to the show.

swyx: One thing that slightly makes me uncomfortable is like, why do we have to build cli? Why can’t we just expose APIs? Like,

Kyle: I, I have, I have an interesting answer to this. So there are a couple reasons.

Like there’s, there’s like, you know, portability is like one issue. Like, you know, like sometimes APIs are not like discoverable or like reachable by, by some, you know, types of [01:08:00] things. There’s some element of locality, right? Like, uh, like the CLI is like literally you interfacing with your like local system, which is a little bit different.

You could still do it by API, but like there’s this highlighting of like, what is the difference between like a CLI and an MCP, right? Like they kind of occupy the same purposes and you call them, it does something on the system and, and that’s done. I think that in pre-training there’s just an enormous amount.

Oh, okay. Command line data. Yeah.

Yeah. Like e even let’s ignore our, let’s let’s ignore our l Like you’re doing no harness, you’re doing no harness push training. Just the amount of like CLI versus API documentation for just like navigating this world of the CLI in your file system through that is just enormous.

Nader: Yeah. Yeah.

Kyle: Right. I

Nader: think there’s a, there’s a couple of things too. Like if, let’s say we wanna, so one I think your intuition’s, right? The CLI is just wrapping the API,

swyx: right? So functional

Nader: functionally, right? Yeah. And I think it’s nice because one, you’re, you’re being very, uh, specific and pedantic even, um, of what and that’s really good ‘cause you’re describing the problem space.

So you know what the, I don’t [01:09:00] know. I don’t wanna call it like what the, the space for vulnerability. You know what network calls you’re making, it’s not arbitrary and that’s not decided on the fly. That’s like pre-decided, which is important from a security perspective. But then if you were to write a bunch of API requests, you would probably do that.

I don’t know. Would the model like use Python to do so? I kind of like that. Everything like a CLI is just dash because it’s ubiquitous. Like it’s just there. And you don’t have to make sure that there’s certain environment variables that are set up. Like if your Python versions, if the My Python version we’re using the same model to go do the same thing, is it gonna write like different code?

It probably would. And so it’s kind of like an nice deal work, right? Yeah. Human. Yeah. No, I think just like making those decisions happen ahead of time versus yeah.

swyx: One last thing on this sort of agent, I guess maybe co-location or whatever you call it, uh, one pattern on tracking for this year, I always try to think about what’s the theme of this year gonna be last year?

Definitely coding agents this year is definitely coding agents, breaking out of containment into broadening third world. I go Definitely has. So

Vibhu: you rent a human?

swyx: Yeah. Yeah.

I’m on here.

swyx: Are you really? [01:10:00]

I’m like $5,000. I’ll do anything. Really? I think so. I need, uh,

swyx: my, uh, my borrow from Costco.

Uh, but I think the best part is only the agent can book me, you know?

Yeah.

swyx: It’s very

Kyle: usually like,

swyx: it’s just like another labor marketplace at Mechanical Turk was this.

So definitely I have a weird story with why I did it. So back to your example of just giving agent access to compute, right? Yeah. You guys are GPU Rich at Nvidia. Yeah, I hooked up.

Nader: He’s not shy about it.

Local GPUs And Scaling Inference

I have, I have a 24 7 agent running, I hooked up to run pot.

It doesn’t shut down instances. And I’m like, I’ve tried prompting you, I’ve given the instruction. Shut down when you’re done. It’s like I to keep it warm, I’ll need it soon. And it’s horrible on time estimates too, ‘cause like they realize it’s like. Yeah, I’ll need it in 45 minutes. 45 minutes, I’ll shut it down.

45 minutes of human time is actually three minute of agent time, so it’s like I’m booting it up, I’m waiting, I’ll just leave it on all night. And mo moo’s good at shutting down after something activity. I had it on my local server, like a little dual GPU thing. It just stays on. I have a little space heater at home now, but careful.

[01:11:00] So basically, you know, they don’t care about the concept of money just burn it. I need it. It’s useful.

Nader: And another DGX spark will be really nice. Like, I, I think I’m looking at it as super useful for agents because Yeah, you buy it once you plug it in and they it can rip. I’m gonna make a, I’m gonna make an Nvidia ad here.

Kyle: Okay. The Blackwell, like RTX 6,000 cards. Pro Pro only, like, I think it’s $8,000. Slightly cheaper. Yeah. Well, it’s much, it’s much cheaper than the data center cards.

Vibhu: Yeah.

Kyle: And it’s got 96 gigabytes of u gram. So if you and your, your crew want to go, like, run a local agent for you, you know, you, you in the home.

I feel like, hmm. It’s got a significant amount of vra m I’ve thought about purchasing this and running in my basement, except my neighbors would hate me.

It’s just a single, like two, three slot. GPU. It’s mostly,

Kyle: yeah, it’s A-V-C-I-E.

Yeah, it’s

Kyle: UCI u. So GPU, you can go by that. I mean, the big difference against like the RTX, like gaming, GPUs, it, I mean, obviously it’s like blackball Pro, like it’s a pro GPU and it has a [01:12:00] lot of E round, which means you can run pretty large models on it.

You can stack four of them for the Maxim Q in a system that’s a beast.

Kyle: It’s beefy. You can run, uh, what is that, 96 ger or anything? 96, uh, you’re on a loge.

Uh, but also they, they are slow. They’re not, I mean, performance of speed will be somewhat slower compared to API like,

Kyle: oh yeah, that, that’s true. So again, the big learning economy of scale allows you to do things that allow you to get both speed and throughput.

Like you can run. I’ll give you an example. There’s an optimization called Wide ep. I’m not gonna go into it fully, but like it featured heavily in, in inference Maxim for Deep seek. And there’s a, there’s a great set of stories from Nvidia and from semi analysis about like why y EP is important, but for like MOE models, it’s like basically essential and you run it like the A Level app parallelism, the level scale up parallelism used for it is like 32.

So it goes beyond that eight barrier. And it like really, really, really is important to have that M mbl, L [01:13:00] 72, GB 200 MD link to serve at scale. And like, it’s like, I don’t remember the, the, you know, cost improvement I think against Hopper, right? Against Hopper. With this MBL L 72 system, you’re getting like 35 times cheaper per token for like a lot of the curve.

Yeah. Which is crazy.

swyx: Yeah.

Kyle: And Normalize per GPU obviously because the part of the GP is cost or the code, the GST part of the cost.

swyx: One thing I’m exploring is the sort of, this year is also the year at the subagent, um, where you have the main agent, but then that also kicks off tools, which are in themselves, agents that have limiteds.

Yeah. And sort of context locally, whatever, right? Yeah. Different prompts. So for example, one thing that Ian does is before you kick off a search, they do like a fast context model where you kick off April or you just to search, uh, across the code base plus all that. That is better than indexing. A a lot of the times, not, not all the times, and, uh, you should sell index for some picks, but like the idea that agents should be able to command subagent and probably run [01:14:00] them like maybe close to inference as well.

I don’t know if that’s like architecturally possible or even

Kyle: Yeah, we’re, we’re thinking about that for dmo. That’s like our big theme for the year,

swyx: because like you, like if you can design that into your stuff, then a lot of people, a lot more people will use it. Right now it’s like just kind of theoretical because.

You do pay a lot of like back and forth, uh, coordination costs. Yes.

Vibhu: I think it’ll net speed up though, right? Like even at a basic level, speculative decoding, you’re running a small model, you’re running two instances, but it’s not,

swyx: that is one example. Yes.

Kyle: Yeah. But this is like a little bit like different with like agents.

Agents, yeah. This is not spec. I think, I think there’s like a summarization of that trend that I like to do or I like to say to my team, it’s like, this is the year. So there are two things. This is the year system as model, right? Where like instead of having like a single model be a thing, you have a system of models and components that are working together to like emulate the black box model.

So when you, when you make an API call to something that’s like, like a multi-agent in the background, it still looks like an API called a model. You’re still getting back to

swyx: grants, but under the hood.

Kyle: Yeah, under the hood. It’s like a [01:15:00] billion different models. And that’s a lot of complexity, with Dynamo and with other libraries and media we’re, we’re looking to help manage

Nader: that complaint.

Yeah. It’s funny because we actually, for CES, we just released the model router. Uh, for DGX Spark where you can have a local model that’s running on the spark and then also a foundational model and then the model router decides when to send queries to which one. So it’s no longer this like either or.

It’s used the best stuff for everything that’s available to you. You have a good post-training bottle that’s running on

swyx: these. There are leads that are also the bread functionality of being able to manage the spark.

Kyle: Oh, that’d be cool. Oh yeah,

swyx: I did be able feature request. There we go.

Long Running Agents And SF Reflections

Kyle: I actually like a question, like I, I like to like extend and flip over.

How much longer do you guys think like agents are gonna be running? Because that’s one thing I’ve been throwing around, like, what happens when, I

mean always are

Kyle: it

even affects the, like back to the prefilled d the decode, right? Like, yeah. Codex is, I’d say, compared to cloud code, it’s much longer at tasks like, yeah, that thing, we’ll, like to run 6, 7, 8 hours.

I’ll run it overnight.

Kyle: Yeah.

And I’ll, I’ll go back and I have like a little crappy logging software I use and there’s just times where it wants to, like, I’m gonna go deep on [01:16:00] research and it’ll, I eat up 80,000 tokens go on another go on another, yeah. Just eat through tokens and you know, that’s part of it.

Like, at the end it does, it does hit a long task. And I think you only see that, that expense. Yeah.

Nader: I, yeah, there’s insatiable demand for tokens and every improvement that comes kind of just makes our demand even higher. It’s kind of funny, right? Like if you have like a teammate and you ask me to do a task and they’re like, should I save some effort and not think too hard about this task?

I’m like, f**k no.

I mean, my favorite was like, you can, you can have four shots, right? Yeah. Like the original codex before the app. You, why do one call, like, give it four attempts? Just, just use all the token to out, right? Try Moreal try, try again. Try more. It’s

Kyle: like, it’s like the, the meta index right?

Is the thing that tracks like how long models are able to run. I expect that we’ll just see like log linear, if not log super linear growth. We will see before the end of the year an agent that is capable of running for longer than 24 hours with like self consistency the entire time.

I, I would also poke at different domains, having different [01:17:00] desires, right?

Like at a consumer level. I’m getting slightly frustrated at 20 minutes per basic query. Sure. You can optimize, you know, six, eight hour. I don’t see myself shooting off many one week agents. Right. Someone doing like, okay, GPU kernel research or medical or biological, like, you know, in, in those domains Sure.

Shoot off a lot. That take a, so like I think it will be somewhat domain specific ‘cause you also really need to turn that in. Right.

Kyle: It’s funny one, those was doing your taxes. Right. Like, that’s tax. Yeah, that’s, yeah. Okay. Yeah.

Nader: Get it right. I wonder if like this major school say sort of like, uh, speculative decoding is like your agent figuring out what you might be prompting it the next day at night and like pre fetching.

swyx: Yeah, you can do

that.

Nader: Yeah. Really? Branch, branch prediction.

swyx: Oh, well no, that, well, that’s, that’s too, that’s too low level, but yes. Sorry. Yeah, yeah, yeah. One question I gotta get, so like, uh, we actually did record a part with the, the beat folks. Uh, with Sarah right here, their chart is the human equivalent work, uh, hours of work rather than how long it has themselves are, are being [01:18:00] autonomous.

And that, that’s a huge difference, right? Like human work, five hours agent work, 30 minutes, like it’s actually 30 minutes not, uh, yeah. Firearms, right? Like, so like that, that, that chart that you see is them estimating what the human equivalent replacement is. Um, I think the, I think actually Enro release a more recent chart.

That showed cloud code autonomy from their production traffic numbers, and that was 20 to 45 minutes. That’s roughly where we are. So yeah. Yeah, that’s the sort of realistic thing. I mean, I, I do think like there’s experimental setups we can just like, Ralph with and like just prompt it to keep going, uh, when it stops.

And obviously you can, that can go arbitrarily long,

Nader: I feel like

from my

Nader: experience. Yeah. I guess 20 to 40 minutes seems right for when I’m using like Codex or cloud code. But then like what, I always try to just, like, if I wanna spin up like a new, there’s a net new project, I’ll, I’ll often start to rep it and like it’ll end for I believe, yeah, yeah.

Like spin up like the, their new, like from the V three agent. Like it’ll spin up a web browser and like click around and discover new bugs and just keep churning. Um, so I, I think like my longest was like over an hour that, hey, I’ve been churning

I think before [01:19:00] we see super long running. I think there’s gonna be a bit of an efficiency hit.

So. Sure you can take an hour and go down paths, but you also want you wanna be more efficient, you wanna be smarter in your reasoning, right? So I think that’ll actually go down before we go back up. Like, you don’t wanna scale non-optimized systems just for the heck of it. As much as I love saying, use all the tokens, um, you know, they are expensive.

Like going from dance to reasoning models, that’s an added cost, right? You’re paying for a lot of tokens and it doesn’t make sense to just scale stuff that’s not optimized. So there’s, there’s always that little balance.

Nader: Yeah.

But you know. I think you’ll see both sides of it.

Nader: Yeah. So 2023 was super exciting.

I think if you were in SF you were like, okay, uh, I know this is gonna be a huge world changing moment, but it seemed like, you know, no one had known yet. And maybe even before, was it 2022 maybe?

swyx: Yeah, yeah. I would say, yeah, like RU had this tweet where like everyone was in SF from like 2021 to 2023. Yeah.

Understood what it was like to be late, early.

Nader: Totally. Um, yeah, 2021, that’s when I made my first open AI account. Yeah, it went, um, it was crazy. [01:20:00] And I remember it was so funny ‘cause at the time SF had not been doing well. So pretty much what it felt like was the concentration of founders in the city had ro had risen because, um, where my neighbors were used to doing a bunch of stuff, those people had all left.

So the only people that were still in the city were people that really wanted to build It was cheap tech. It was, yeah. It was also way cheaper. I feel really bad anyone, uh, who is trying to get rent now, but there was, uh, cell was they had a huge office.

swyx: So blockchain in Yeah, like took over the, the old Casper building.

Nader: Yeah. They had the showroom and they had the, like the, what would, I think it was like the back warehouse. It was, and it was a huge office. And

swyx: it’s right across an opening Eyes in New Link.

Nader: Yeah. It was in

the original arena.

swyx: I named the Arena because of it.

Nader: Yeah. Yeah. And so it was really exciting because like vo flow I think uh, I forgot the Minify.

Yeah. Minify, uh, brev was there. You guys were there. I remember. That was actually, it was there that you bought the AI engineer domain.

swyx: Yeah. I didn’t know what I was gonna do in ai. I, I wanna do something,

Nader: but it was kind of this, it was a really fun moment where we were kind of all in this solo space and it, um, I don’t know.

It was, [01:21:00] it was a really cool community, especially being so

swyx: early. Yeah. And so it, then you got me early cruise access. Oh yeah. So there was a going period of time. They both cruises and Waymo’s were just free. Yeah, always.

If you had, I mean, they’re, they’re so Back Cell is opened again.

swyx: Yeah. So Nature Zoo.

Zoo is Nature Zoo. Zoo Robot Taxi. Yeah. So Totally. Yeah.

Nader: Oh. But yeah. And so it’s actually really cool that you guys have this studio so close to, uh, cell. Yeah. This rock climbing gin right around the corner. It was like, um, 2000. Oh yeah. Yeah. It’s, it’s an awesome block.

swyx: Cool. Yeah. Just, and you bit services partnership.

Uh, I do think one, one thing I try to do with the podcast is like bring, like what is, I get to be a San Francisco to the rest of the world and also just like. Maybe give, uh, yeah.

Nader: Yeah. My favorite talk was in the city, uh, and

swyx: yeah, stick and stream. I know. It’s very good.

Nader: Yeah. And I guess what it’s like to be in San Francisco I think is just everyone seems to be super supportive.

Uh, sometimes I feel like the city believes in you more than you do. And even, uh, I don’t know if you remember, but I remember [01:22:00] posting my first blog post and I had met you on Twitter and you gave me like an hour of your time super randomly, and you kind of coached me through, uh, writing content for developers.

And I was trying really hard not to come off salesy or plug myself. And so I kind of stripped all personality out of the blog post. Yeah. And you, you brought that out. You’re like, people don’t, it’s, it’s okay to talk about what you’re doing. Like you don’t have to be weird about it. And I remember just that, I think that really helped me kind of figure out what our voice is and not shy away from it.

And so always really grateful for you. Hey, you inject your voice into like, everything. Now it’s actually a huge advantage to be like very

Kyle: genuine about what you care about.

swyx: Yeah. Yeah. You imagine like summer, some infra in DMU and like, it’s like, can you gimme feedback on this blog post? And it’s pretty boring and you’re like.

Find like, you know, he looks interesting. I’ll just do a zoom call and then you meet this guy. Yeah, right. He’s so energetic, so just be right. There’s, but like, I think people are trained to write a certain way in school and Yeah. They never totally see there’s like a broader well,

and

Nader: lots un unlearn

Kyle: writing.

Writing is thinking and like everyone thinks differently. So [01:23:00] like, might as well as just like,

swyx: yeah. Yeah.

Kyle: Write your way.

swyx: Cool. Well, thank you for, uh, in indulging with us, uh, really broad breaking discussion, but I love, like, you guys are like, sort of like the sort of young faces on video with so much energy and, but like also lot of technic death and I think, uh, people learn about for this session.

So thank you.

Nader: This was awesome. Thank you guys. So thank you for everything that you’ve done in the talk. Yeah, NG the podcast, all the above. And uh, C-O-T-C-I really forward to it. Yeah. Cool. Thanks. That’s awesome. Thank you. Thank you.

All speakers are announced at AIE EU, schedule coming soon. Join us there or in Miami with the renowned organizers of React Miami! Singapore CFP also open!

We’ve called this out a few times over in AINews, but the overwhelming consensus in the Valley is that “the IDE is Dead”. In November it was just a gut feeling, but now we actually have data: even at the canonical “VSCode Fork” company, people are officially using more agents than tab autocomplete (the first wave of AI coding):

Cursor has launched cloud agents for a few months now, and this specific launch is around Computer Use, which has come a long way since we first talked with Anthropic about it in 2024, and which Jonas productized as Autotab:

We also take the opportunity to do a live demo, talk about slash commands and subagents, and the future of continual learning and personalized coding models, something that Sam previously worked on at New Computer. (The fact that both of these folks are top tier CEOs of their own startups that have now joined the insane talent density gathering at Cursor should also not be overlooked).

Full Episode on YouTube!

please like and subscribe!

Timestamps

00:00 Agentic Code Experiments00:53 Why Cloud Agents Matter02:08 Testing First Pillar03:36 Video Reviews Second Pillar04:29 Remote Control Third Pillar06:17 Meta Demos and Bug Repro13:36 Slash Commands and MCPs18:19 From Tab to Team Workflow31:41 Minimal Web UI Philosophy32:40 Why No File Editor34:38 Full Stack Cursor Debate36:34 Model Choice and Auto Routing38:34 Parallel Agents and Best Of N41:41 Subagents and Context Management44:48 Grind Mode and Throughput Future01:00:24 Cloud Agent Onboarding and Memory

Transcript

EP 77 - CURSOR - Audio version

[00:00:00]

Agentic Code Experiments

Samantha: This is another experiment that we ran last year and didn’t decide to ship at that time, but may come back to LM Judge, but one that was also agentic and could write code. So it wasn’t just picking but also taking the learnings from two models or and models that it was looking at and writing a new diff.

And what we found was that there were strengths to using models from different model providers as the base level of this process. Basically you could get almost like a synergistic output that was better than having a very unified like bottom model tier.

Jonas: We think that over the coming months, the big unlock is not going to be one person with a model getting more done, like the water flowing faster and we’ll be making the pipe much wider and so paralyzing more, whether that’s swarms of agents or parallel agents, both of those are things that contribute to getting much more done in the same amount of time.

Why Cloud Agents Matter

swyx: This week, one of the biggest launches that Cursor’s ever done is cloud agents. I think you, you had [00:01:00] cloud agents before, but this was like, you give cursor a computer, right? Yeah. So it’s just basically they bought auto tab and then they repackaged it. Is that what’s going on, or,

Jonas: that’s a big part of it.

Yeah. Cloud agents already ran in their own computers, but they were sort of site reading code. Yeah. And those computers were not, they were like blank VMs typically that were not set up for the Devrel X for whatever repo the agents working on. One of the things that we talk about is if you put yourself in the model shoes and you were seeing tokens stream by and all you could do was cite read code and spit out tokens and hope that you had done the right thing,

swyx: no chance

Jonas: I’d be so bad.

Like you obviously you need to run the code. And so that I think also is probably not that contrarian of a take, but no one has done that yet. And so giving the model the tools to onboard itself and then use full computer use end-to-end pixels in coordinates out and have the cloud computer with different apps in it is the big unlock that we’ve seen internally in terms of use usage of this going from, oh, we use it for little copy changes [00:02:00] to no.

We’re really like driving new features with this kind of new type of entech workflow. Alright, let’s see it. Cool.

Live Demo Tour

Jonas: So this is what it looks like in cursor.com/agents. So this is one I kicked off a while ago. So on the left hand side is the chat. Very classic sort of agentic thing. The big new thing here is that the agent will test its changes.

So you can see here it worked for half an hour. That is because it not only took time to write the tokens of code, it also took time to test them end to end. So it started Devrel servers iterate when needed. And so that’s one part of it is like model works for longer and doesn’t come back with a, I tried some things pr, but a I tested at pr that’s ready for your review.

One of the other intuition pumps we use there is if a human gave you a PR asked you to review it and you hadn’t, they hadn’t tested it, you’d also be annoyed because you’d be like, only ask me for a review once it’s actually ready. So that’s what we’ve done with

Testing Defaults and Controls

swyx: simple question I wanted to gather out front.

Some prs are way smaller, [00:03:00] like just copy change. Does it always do the video or is it sometimes,

Jonas: Sometimes.

swyx: Okay. So what’s the judgment?

Jonas: The model does it? So we we do some default prompting with sort. What types of changes to test? There’s a slash command that people can do called slash no test, where if you do that, the model will not test,

swyx: but the default is test.

Jonas: The default is to be calibrated. So we tell it don’t test, very simple copy changes, but test like more complex things. And then users can also write their agents.md and specify like this type of, if you’re editing this subpart of my mono repo, never tested ‘cause that won’t work or whatever.

Videos and Remote Control

Jonas: So pillar one is the model actually testing Pillar two is the model coming back with a video of what it did.

We have found that in this new world where agents can end-to-end, write much more code, reviewing the code is one of these new bottlenecks that crop up. And so reviewing a video is not a substitute for reviewing code, but it is an entry point that is much, much easier to start with than glancing at [00:04:00] some giant diff.

And so typically you kick one off you, it’s done you come back and the first thing that you would do is watch this video. So this is a, video of it. In this case I wanted a tool tip over this button. And so it went and showed me what that looks like in, in this video that I think here, it actually used a gallery.

So sometimes it will build storybook type galleries where you can see like that component in action. And so that’s pillar two is like these demo videos of what it built. And then pillar number three is I have full remote control access to this vm. So I can go heat in here. I can hover things, I can type, I have full control.

And same thing for the terminal. I have full access. And so that is also really useful because sometimes the video is like all you need to see. And oftentimes by the way, the video’s not perfect, the video will show you, is this worth either merging immediately or oftentimes is this worth iterating with to get it to that final stage where I am ready to merge in.

So I can go through some other examples where the first video [00:05:00] wasn’t perfect, but it gave me confidence that we were on the right track and two or three follow-ups later, it was good to go. And then I also have full access here where some things you just wanna play around with. You wanna get a feel for what is this and there’s no substitute to a live preview.

And the VNC kind of VM remote access gives you that.

swyx: Amazing What, sorry? What is VN. And

Jonas: just the remote desktop. Remote desktop. Yeah.

swyx: Sam, any other details that you always wanna call out?

Samantha: Yeah, for me the videos have been super helpful. I would say, especially in cases where a common problem for me with agents and cloud agents beforehand was almost like under specification in my requests where our plan mode and going really back and forth and getting detailed implementation spec is a way to reduce the risk of under specification, but then similar to how human communication breaks down over time, I feel like you have this risk where it’s okay, when I pull down, go to the triple of pulling down and like running this branch locally, I’m gonna see that, like I said, this should be a toggle and you have a checkbox and like, why didn’t you get that detail?

And having the video up front just [00:06:00] has that makes that alignment like you’re talking about a shared artifact with the agent. Very clear, which has been just super helpful for me.

Jonas: I can quickly run through some other Yes. Examples.

Meta Agents and More Demos

Jonas: So this is a very front end heavy one. So one question I was

swyx: gonna say, is this only for front

Jonas: end?

Exactly. One question you might have is this only for front end? So this is another example where the thing I wanted it to implement was a better error message for saving secrets. So the cloud agents support adding secrets, that’s part of what it needs to access certain systems. Part of onboarding that is giving access.

This is cloud is working on

swyx: cloud agents. Yes.

Jonas: So this is a fun thing is

Samantha: it can get super meta. It

Jonas: can get super meta, it can start its own cloud agents, it can talk to its own cloud agents. Sometimes it’s hard to wrap your mind around that. We have disabled, it’s cloud agents starting more cloud agents. So we currently disallow that.

Someday you might. Someday we might. Someday we might. So this actually was mostly a backend change in terms of the error handling here, where if the [00:07:00] secret is far too large, it would oh, this is actually really cool. Wow. That’s the Devrel tools. That’s the Devrel tools. So if the secret is far too large, we.

Allow secrets above a certain size. We have a size limit on them. And the error message there was really bad. It was just some generic failed to save message. So I was like, Hey, we wanted an error message. So first cool thing it did here, zero prompting on how to test this. Instead of typing out the, like a character 5,000 times to hit the limit, it opens Devrel tools, writes js, or to paste into the input 5,000 characters of the letter A and then hit save, closes the Devrel tools, hit save and gets this new gets the new error message.

So that looks like the video actually cut off, but here you can see the, here you can see the screenshot of the of the error message. What, so that is like frontend backend end-to-end feature to, to get that,

swyx: yeah.

Jonas: And

swyx: And you just need a full vm, full computer run everything.

Okay. Yeah.

Jonas: Yeah. So we’ve had versions of this. This is one of the auto tab lessons where we started that in 2022. [00:08:00] No, in 2023. And at the time it was like browser use, DOM, like all these different things. And I think we ended up very sort of a GI pilled in the sense that just give the model pixels, give it a box, a brain in a box is what you want and you want to remove limitations around context and capabilities such that the bottleneck should be the intelligence.

And given how smart models are today, that’s a very far out bottleneck. And so giving it its full VM and having it be onboarded with Devrel X set up like a human would is just been for us internally a really big step change in capability.

swyx: Yeah I would say, let’s call it a year ago the models weren’t even good enough to do any of this stuff.

So

Samantha: even six months ago. Yeah.

swyx: So yeah what people have told me is like round about Sonder four fire is when this started being good enough to just automate fully by pixel.

Jonas: Yeah, I think it’s always a question of when is good enough. I think we found in particular with Opus 4 5, 4, 6, and Codex five three, that those were additional step [00:09:00] changes in the autonomy grade capabilities of the model to just.

Go off and figure out the details and come back when it’s done.

swyx: I wanna appreciate a couple details. One 10 Stack Router. I see it. Yeah. I’m a big fan. Do you know any, I have to name the 10 Stack.

Jonas: No.

swyx: This just a random lore. Some buddy Sue Tanner. My and then the other thing if you switch back to the video.

Jonas: Yeah.

swyx: I wanna shout out this thing. Probably Sam did it. I don’t know

Jonas: the chapters.

swyx: What is this called? Yeah, this is called Chapters. Yeah. It’s like a Vimeo thing. I don’t know. But it’s so nice the design details, like the, and obviously a company called Cursor has to have a beautiful cursor

Samantha: and it is

swyx: the cursor.

Samantha: Cursor.

swyx: You see it branded? It’s the cursor. Cursor, yeah. Okay, cool. And then I was like, I complained to Evan. I was like, okay, but you guys branded everything but the wallpaper. And he was like, no, that’s a cursor wallpaper. I was like, what?

Samantha: Yeah. Rio picked the wallpaper, I think. Yeah. The video.

That’s probably Alexi and yeah, a few others on the team with the chapters on the video. Matthew Frederico. There’s been a lot of teamwork on this. It’s a huge effort.

swyx: I just, I like design details.

Samantha: Yeah.

swyx: And and then when you download it adds like a little cursor. Kind of TikTok clip. [00:10:00] Yes. Yes.

So it’s to make it really obvious is from Cursor,

Jonas: we did the TikTok branding at the end. This was actually in our launch video. Alexi demoed the cloud agent that built that feature. Which was funny because that was an instance where one of the things that’s been a consequence of having these videos is we use best of event where you run head to head different models on the same prompt.

We use that a lot more because one of the complications with doing that before was you’d run four models and they would come back with some giant diff, like 700 lines of code times four. It’s what are you gonna do? You’re gonna review all that’s horrible. But if you come back with four 22nd videos, yeah, I’ll watch four 22nd videos.

And then even if none of them is perfect, you can figure out like, which one of those do you want to iterate with, to get it over the line. Yeah. And so that’s really been really fun.

Bug Repro Workflow

Jonas: Here’s another example. That’s we found really cool, which is we’ve actually turned since into a slash command as well slash [00:11:00] repro, where for bugs in particular, the model of having full access to the to its own vm, it can first reproduce the bug, make a video of the bug reproducing, fix the bug, make a video of the bug being fixed, like doing the same pattern workflow with obviously the bug not reproducing.

And that has been the single category that has gone from like these types of bugs, really hard to reproduce and pick two tons of time locally, even if you try a cloud agent on it. Are you confident it actually fixed it to when this happens? You’ll merge it in 90 seconds or something like that.

So this is an example where, let me see if this is the broken one or the, okay, this is the fixed one. Okay. So we had a bug on cursor.com/agents where if you would attach images where remove them. Then still submit your prompt. They would actually still get attached to the prompt. Okay. And so here you can see Cursor is using, its full desktop by the way.

This is one of the cases where if you just do, browse [00:12:00] use type stuff, you’ll have a bad time. ‘cause now it needs to upload files. Like it just uses its native file viewer to do that. And so you can see here it’s uploading files. It’s going to submit a prompt and then it will go and open up. So this is the meta, this is cursor agent, prompting cursor agent inside its own environment.

And so you can see here bug, there’s five images attached, whereas when it’s submitted, it only had one image.

swyx: I see. Yeah. But you gotta enable that if you’re gonna use cur agent inside cur.

Jonas: Exactly. And so here, this is then the after video where it went, it does the same thing. It attaches images, removes, some of them hit send.

And you can see here, once this agent is up, only one of the images is left in the attachments. Yeah.

swyx: Beautiful.

Jonas: Okay. So easy merge.

swyx: So yeah. When does it choose to do this? Because this is an extra step.

Jonas: Yes. I think I’ve not done a great job yet of calibrating the model on when to reproduce these things.

Yeah. Sometimes it will do it of its own accord. Yeah. We’ve been conservative where we try to have it only do it when it’s [00:13:00] quite sure because it does add some amount of time to how long it takes it to work on it. But we also have added things like the slash repro command where you can just do, fix this bug slash repro and then it will know that it should first make you a video of it actually finding and making sure it can reproduce the bug.

swyx: Yeah. Yeah. One sort of ML topic this ties into is reward hacking, where while you write test that you update only pass. So first write test, it shows me it fails, then make you test pass, which is a classic like red green.

Jonas: Yep.

swyx: Like

Jonas: A-T-D-D-T-D-D

swyx: thing.

No, very cool. Was that the last demo? Is there

Jonas: Yeah.

Anything I missed on the demos or points that you think? I think that

Samantha: covers it well. Yeah.

swyx: Cool. Before we stop the screen share, can you gimme like a, just a tour of the slash commands ‘cause I so God ready. Huh, what? What are the good ones?

Samantha: Yeah, we wanna increase discoverability around this too.

I think that’ll be like a future thing we work on. Yeah. But there’s definitely a lot of good stuff now

Jonas: we have a lot of internal ones that I think will not be that interesting. Here’s an internal one that I’ve made. I don’t know if anyone else at Cursor uses this one. Fix bb.

Samantha: I’ve never heard of it.

Jonas: Yeah.[00:14:00]

Fix Bug Bot. So this is a thing that we want to integrate more tightly on. So you made it for

swyx: yourself.

Jonas: I made this for myself. It’s actually available to everyone in the team, but yeah, no one knows about it. But yeah, there will be Bug bot comments and so Bug Bot has a lot of cool things. We actually just launched Bug Bot Auto Fix, where you can click a button and or change a setting and it will automatically fix its own things, and that works great in a bunch of cases.

There are some cases where having the context of the original agent that created the PR is really helpful for fixing the bugs, because it might be like, oh, the bug here is that this, is a regression and actually you meant to do something more like that. And so having the original prompt and all of the context of the agent that worked on it, and so here I could just do, fix or we used to be able to do fixed PB and it would do that.

No test is another one that we’ve had. Slash repro is in here. We mentioned that one.

Samantha: One of my favorites is cloud agent diagnosis. This is one that makes heavy use of the Datadog MCP. Okay. And I [00:15:00] think Nick and David on our team wrote, and basically if there is a problem with a cloud agent we’ll spin up a bunch of subs.

Like a single

swyx: instance.

Samantha: Yeah. We’ll take the ideas and argument and spin up a bunch of subagents using the Datadog MCP to explore the logs and find like all of the problems that could have happened with that. It takes the debugging time, like from potentially you can do quick stuff quickly with the Datadog ui, but it takes it down to, again, like a single agent call as opposed to trolling through logs yourself.

Jonas: You should also talk about the stuff we’ve done with transcripts.

Samantha: Yes. Also so basically we’ve also done some things internally. There’ll be some versions of this as we ship publicly soon, where you can spit up an agent and give it access to another agent’s transcript to either basically debug something that happened.

So act as an external debugger. I see. Or continue the conversation. Almost like forking it.

swyx: A transcript includes all the chain of thought for the 11 minutes here. 45 minutes there.

Samantha: Yeah. That way. Exactly. So basically acting as a like secondary agent that debugs the first, so we’ve started to push more and

swyx: they’re all the same [00:16:00] code.

It is just the different prompts, but the sa the same.

Samantha: Yeah. So basically same cloud agent infrastructure and then same harness. And then like when we do things like include, there’s some extra infrastructure that goes into piping in like an external transcript if we include it as an attachment.

But for things like the cloud agent diagnosis, that’s mostly just using the Datadog MCP. ‘Cause we also launched CPS along with along with this cloud agent launch, launch support for cloud agent cps.

swyx: Oh, that was drawn out.

Jonas: We won’t, we’ll be doing a bigger marketing moment for it next week, but, and you can now use CPS and

swyx: People will listen to it as well.

Yeah,

Jonas: they’ll

Samantha: be ahead of the third. They’ll be ahead. And I would I actually don’t know if the Datadog CP is like publicly available yet. I realize this not sure beta testing it, but it’s been one of my favorites to use. So

swyx: I think that one’s interesting for Datadog. ‘cause Datadog wants to own that site.

Interesting with Bits. I don’t know if you’ve tried bits.

Samantha: I haven’t tried bits.

swyx: Yeah.

Jonas: That’s their cloud agent

swyx: product. Yeah. Yeah. They want to be like we own your logs and give us our, some part of the, [00:17:00] self-healing software that everyone wants. Yeah. But obviously Cursor has a strong opinion on coding agents and you, you like taking away from the which like obviously you’re going to do, and not every company’s like Cursor, but it’s interesting if you’re a Datadog, like what do you do here?

Do you expose your logs to FDP and let other people do it? Or do you try to own that it because it’s extra business for you? Yeah. It’s like an interesting one.

Samantha: It’s a good question. All I know is that I love the Datadog MCP,

Jonas: And yeah, it is gonna be no, no surprise that people like will demand it, right?

Samantha: Yeah.

swyx: It’s, it’s like any

system

swyx: of record company like this, it’s like how much do you give away? Cool. I think that’s that for the sort of cloud agents tour. Cool. And we just talk about like cloud agents have been when did Kirsten loves cloud agents? Do you know, in June

Jonas: last year.

swyx: June last year. So it’s been slowly develop the thing you did, like a bunch of, like Michael did a post where himself, where he like showed this chart of like ages overtaking tap. And I’m like, wow, this is like the biggest transition in code.

Jonas: Yeah.

swyx: Like in, in [00:18:00] like the last,

Jonas: yeah. I think that kind of got turned out.

Yeah. I think it’s a very interest,

swyx: not at all. I think it’s been highlighted by our friend Andre Kati today.

Jonas: Okay.

swyx: Talk more about it. What does it mean? Yeah. Is I just got given like the cursor tab key.

Jonas: Yes. Yes.

swyx: That’s that’s

Samantha: cool.

swyx: I know, but it’s gonna be like put in a museum.

Jonas: It is.

Samantha: I have to say I haven’t used tab a little bit myself.

Jonas: Yeah. I think that what it looks like to code with AI code generally creates software, even if you want to go higher level. Is changing very rapidly. No, not a hot take, but I think from our vendor’s point at Cursor, I think one of the things that is probably underappreciated from the outside is that we are extremely self-aware about that fact and Kerscher, got its start in phase one, era one of like tab and auto complete.

And that was really useful in its time. But a lot of people start looking at text files and editing code, like we call it hand coding. Now when you like type out the actual letters, it’s

swyx: oh that’s cute.

Jonas: Yeah.

swyx: Oh that’s cute.

Jonas: You’re so boomer. So boomer. [00:19:00] And so that I think has been a slowly accelerating and now in the last few months, rapidly accelerating shift.

And we think that’s going to happen again with the next thing where the, I think some of the pains around tab of it’s great, but I actually just want to give more to the agent and I don’t want to do one tab at a time. I want to just give it a task and it goes off and does a larger unit of work and I can.

Lean back a little bit more and operate at that higher level of abstraction that’s going to happen again, where it goes from agents handing you back diffs and you’re like in the weeds and giving it, 32nd to three minute tasks, to, you’re giving it, three minute to 30 minute to three hour tasks and you’re getting back videos and trying out previews rather than immediately looking at diffs every single time.

swyx: Yeah. Anything to add?

Samantha: One other shift that I’ve noticed as our cloud agents have really taken off internally has been a shift from primarily individually driven development to almost this collaborative nature of development for us, slack is actually almost like a development on [00:20:00] Id basically.

So I

swyx: like maybe don’t even build a custom ui, like maybe that’s like a debugging thing, but actually it’s that.

Samantha: I feel like, yeah, there’s still so much to left to explore there, but basically for us, like Slack is where a lot of development happens. Like we will have these issue channels or just like this product discussion channels where people are always at cursing and that kicks off a cloud agent.

And for us at least, we have team follow-ups enabled. So if Jonas kicks off at Cursor in a thread, I can follow up with it and add more context. And so it turns into almost like a discussion service where people can like collaborate on ui. Oftentimes I will kick off an investigation and then sometimes I even ask it to get blame and then tag people who should be brought in. ‘cause it can tag people in Slack and then other people will come

swyx: in, can tag other people who are not involved in conversation. Yes. Can just do at Jonas if say, was talking to,

Samantha: yeah.

swyx: That’s cool. You should, you guys should make a big good deal outta that.

Samantha: I know. It’s a lot to, I feel like there’s a lot more to do with our slack surface area to show people externally. But yeah, basically like it [00:21:00] can bring other people in and then other people can also contribute to that thread and you can end up with a PR again, with the artifacts visible and then people can be like, okay, cool, we can merge this.

So for us it’s like the ID is almost like moving into Slack in some ways as well.

swyx: I have the same experience with, but it’s not developers, it’s me. Designer salespeople.

Samantha: Yeah.

swyx: So me on like technical marketing, vision, designer on design and then salespeople on here’s the legal source of what we agreed on.

And then they all just collaborate and correct. The agents,

Jonas: I think that we found when these threads is. The work that is left, that the humans are discussing in these threads is the nugget of what is actually interesting and relevant. It’s not the boring details of where does this if statement go?

It’s do we wanna ship this? Is this the right ux? Is this the right form factor? Yeah. How do we make this more obvious to the user? It’s like those really interesting kind of higher order questions that are so easy to collaborate with and leave the implementation to the cloud agent.

Samantha: Totally. And no more discussion of am I gonna do this? Are you [00:22:00] gonna do this cursor’s doing it? You just have to decide. You like it.

swyx: Sometimes the, I don’t know if there’s a, this probably, you guys probably figured this out already, but since I, you need like a mute button. So like cursor, like we’re going to take this offline, but still online.

But like we need to talk among the humans first. Before you like could stop responding to everything.

Jonas: Yeah. This is a design decision where currently cursor won’t chime in unless you explicitly add Mention it. Yeah. Yeah.

Samantha: So it’s not always listening.

Yeah.

Jonas: I can see all the intermediate messages.

swyx: Have you done the recursive, can cursor add another cursor or spawn another cursor?

Samantha: Oh,

Jonas: we’ve done some versions of this.

swyx: Because, ‘cause it can add humans.

Jonas: Yes. One of the other things we’ve been working on that’s like an implication of generating the code is so easy is getting it to production is still harder than it should be.

And broadly, you solve one bottleneck and three new ones pop up. Yeah. And so one of the new bottlenecks is getting into production and we have a like joke internally where you’ll be talking about some feature and someone says, I have a PR for that. Which is it’s so easy [00:23:00] to get to, I a PR for that, but it’s hard still relatively to get from I a PR for that to, I’m confident and ready to merge this.

And so I think that over the coming weeks and months, that’s a thing that we think a lot about is how do we scale up compute to that pipeline of getting things from a first draft An agent did.

swyx: Isn’t that what Merge isn’t know what graphite’s for, like

Jonas: graphite is a big part of that. The cloud agent testing

swyx: Is it fully integrated or still different companies

Jonas: working on I think we’ll have more to share there in the future, but the goal is to have great end-to-end experience where Cursor doesn’t just help you generate code tokens, it helps you create software end-to-end.

And so review is a big part of that, that I think especially as models have gotten much better at writing code, generating code, we’ve felt that relatively crop up more,

swyx: sorry this is completely unplanned, but like there I have people arguing one to you need ai. To review ai and then there is another approach, thought school of thought where it’s no, [00:24:00] reviews are dead.

Like just show me the video. It’s it like,

Samantha: yeah. I feel again, for me, the video is often like alignment and then I often still wanna go through a code review process.

swyx: Like still look at the files and

Samantha: everything. Yeah. There’s a spectrum of course. Like the video, if it’s really well done and it does like fully like test everything, you can feel pretty competent, but it’s still helpful to, to look at the code.

I make hep pay a lot of attention to bug bot. I feel like Bug Bot has been a great really highly adopted internally. We often like, won’t we tell people like, don’t leave bug bot comments unaddressed. ‘cause we have such high confidence in it. So people always address their bug bot comments.

Jonas: Once you’ve had two cases where you merged something and then you went back later, there was a bug in it, you merged, you went back later and you were like, ah, bug Bot had found that I should have listened to Bug Bot.

Once that happens two or three times, you learn to wait for bug bot.

Samantha: Yeah. So I think for us there’s like that code level review where like it’s looking at the actual code and then there’s like the like feature level review where you’re looking at the features. There’s like a whole number of different like areas.

There’ll probably eventually be things like performance level review, security [00:25:00] review, things like that where it’s like more more different aspects of how this feature might affect your code base that you want to potentially leverage an agent to help with.

Jonas: And some of those like bug bot will be synchronous and you’ll typically want to wait on before you merge.

But I think another thing that we’re starting to see is. As with cloud agents, you scale up this parallelism and how much code you generate. 10 person startups become, need the Devrel X and pipelines that a 10,000 person company used to need. And that looks like a lot of the things I think that 10,000 person companies invented in order to get that volume of software to production safely.

So that’s things like, release frequently or release slowly, have different stages where you release, have checkpoints, automated ways of detecting regressions. And so I think we’re gonna need stacks merg stack diffs merge queues. Exactly. A lot of those things are going to be important

swyx: forward with.

I think the majority of people still don’t know what stack stacks are. And I like, I have many friends in Facebook and like I, I’m pretty friendly with graphite. I’ve just, [00:26:00] I’ve never needed it ‘cause I don’t work on that larger team and it’s just like democratization of no, only here’s what we’ve already worked out at very large scale and here’s how you can, it benefits you too.

Like I think to me, one of the beautiful things about GitHub is that. It’s actually useful to me as an individual solo developer, even though it’s like actually collaboration software.

Jonas: Yep.

swyx: And I don’t think a lot of Devrel tools have figured that out yet. That transition from like large down to small.

Jonas: Yeah. Kers is probably an inverse story.

swyx: This is small down to

Jonas: Yeah. Where historically Kers share, part of why we grew so quickly was anyone on the team could pick it up and in fact people would pick it up, on the weekend for their side project and then bring it into work. ‘cause they loved using it so much.

swyx: Yeah.

Jonas: And I think a thing that we’ve started working on a lot more, not us specifically, but as a company and other folks at Cursor, is making it really great for teams and making it the, the 10th person that starts using Cursor in a team. Is immediately set up with things like, we launched Marketplace recently so other people can [00:27:00] configure what CPS and skills like plugins.

So skills and cps, other people can configure that. So that my cursor is ready to go and set up. Sam loves the Datadog, MCP and Slack, MCP you’ve also been using a lot but

Samantha: also pre-launch, but I feel like it’s so good.

Jonas: Yeah, my cursor should be configured if Sam feels strongly that’s just amazing and required.

swyx: Is it automatically shared or you have to go and.

Jonas: It depends on the MCP. So some are obviously off per user. Yeah. And so Sam can’t off my cursor with my Slack MCP, but some are team off and those can be set up by admins.

swyx: Yeah. Yeah. That’s cool. Yeah, I think, we had a man on the pod when cursor was five people, and like everyone was like, okay, what’s the thing?

And then it’s usually something teams and org and enterprise, but it’s actually working. But like usually at that stage when you’re five, when you’re just a vs. Code fork it’s like how do you get there? Yeah. Will people pay for this? People do pay for it.

Jonas: Yeah. And I think for cloud agents, we expect.[00:28:00]

To have similar kind of PLG things where I think off the bat we’ve seen a lot of adoption with kind of smaller teams where the code bases are not quite as complex to set up. Yes. If you need some insane docker layer caching thing for builds not to take two hours, that’s going to take a little bit longer for us to be able to support that kind of infrastructure.

Whereas if you have front end backend, like one click agents can install everything that they need themselves.

swyx: This is a good chance for me to just ask some technical sort of check the box questions. Can I choose the size of the vm?

Jonas: Not yet. We are planning on adding that. We

swyx: have, this is obviously you want like LXXL, whatever, right?

Like it’s like the Amazon like sort menu.

Jonas: Yes, exactly. We’ll add that.

swyx: Yeah. In some ways you have to basically become like a EC2, almost like you rent a box.

Jonas: You rent a box. Yes. We talk a lot about brain in a box. Yeah. So cursor, we want to be a brain in a box,

swyx: but is the mental model different? Is it more serverless?

Is it more persistent? Is. Something else.

Samantha: We want it to be a bit persistent. The desktop should be [00:29:00] something you can return to af even after some days. Like maybe you go back, they’re like still thinking about a feature for some period of time. So the

swyx: full like sus like suspend the memory and bring it back and then keep going.

Samantha: Exactly.

swyx: That’s an interesting one because what I actually do want, like from a manna and open crawl, whatever, is like I want to be able to log in with my credentials to the thing, but not actually store it in any like secret store, whatever. ‘cause it’s like this is the, my most sensitive stuff.

Yeah. This is like my email, whatever. And just have it like, persist to the image. I don’t know how it was hood, but like to rehydrate and then just keep going from there. But I don’t think a lot of infra works that way. A lot of it’s stateless where like you save it to a docker image and then it’s only whatever you can describe in a Docker file and that’s it.

That’s the only thing you can cl multiple times in parallel.

Jonas: Yeah. We have a bunch of different ways of setting them up. So there’s a dockerfile based approach. The main default way is actually snapshotting

swyx: like a Linux vm

Jonas: like vm, right? You run a bunch of install commands and then you snapshot more or less the file system.

And so that gets you set up for everything [00:30:00] that you would want to bring a new VM up from that template basically.

swyx: Yeah.

Jonas: And that’s a bit distinct from what Sam was talking about with the hibernating and re rehydrating where that is a full memory snapshot as well. So there, if I had like the browser open to a specific page and we bring that back, that page will still be there.

swyx: Was there any discussion internally and just building this stuff about every time you shoot a video it’s actually you show a little bit of the desktop and the browser and it’s not necessary if you just show the browser. If, if you know you’re just demoing a front end application.

Why not just show the browser, right? Like it Yeah,

Samantha: we do have some panning and zooming. Yeah. Like it can decide that when it’s actually recording and cutting the video to highlight different things. I think we’ve played around with different ways of segmenting it and yeah. There’s been some different revs on it for sure.

Jonas: Yeah. I think one of the interesting things is the version that you see now in cursor.com actually is like half of what we had at peak where we decided to unshift or unshipped quite a few things. So two of the interesting things to talk about, one is directly an answer to your [00:31:00] question where we had native browser that you would have locally, it was basically an iframe that via port forwarding could load the URL could talk to local host in the vm.

So that gets you basically, so in

swyx: your machine’s browser,

like

Jonas: in your local browser? Yeah. You would go to local host 4,000 and that would get forwarded to local host 4,000 in the VM via port forward. We unshift that like at

swyx: Eng Rock.

Jonas: Like an Eng Rock. Exactly. We unshift that because we felt that the remote desktop was sufficiently low latency and more general purpose.

So we build Cursor web, but we also build Cursor desktop. And so it’s really useful to be able to have the full spectrum of things. And even for Cursor Web, as you saw in one of the examples, the agent was uploading files and like I couldn’t upload files and open the file viewer if I only had access to the browser.

And we’ve thought a lot about, this might seem funny coming from Cursor where we started as this, vs. Code Fork and I think inherited a lot of amazing things, but also a lot [00:32:00] of legacy UI from VS Code.

Minimal Web UI Surfaces

Jonas: And so with the web UI we wanted to be very intentional about keeping that very minimal and exposing the right sum of set of primitive sort of app surfaces we call them, that are shared features of that cloud.

Environment that you and the agent both use. So agent uses desktop and controls it. I can use desktop and controlled agent runs terminal commands. I can run terminal commands. So that’s how our philosophy around it. The other thing that is maybe interesting to talk about that we unshipped is and we may, both of these things we may reship and decide at some point in the future that we’ve changed our minds on the trade offs or gotten it to a point where, put

swyx: it out there.

Let users tell you they want it. Exactly. Alright, fine.

Why No File Editor

Jonas: So one of the other things is actually a files app. And so we used to have the ability at one point during the process of testing this internally to see next to, I had GID desktop and terminal on the right hand side of the tab there earlier to also have a files app where you could see and edit files.

And we actually felt that in some [00:33:00] ways, by restricting and limiting what you could do there, people would naturally leave more to the agent and fall into this new pattern of delegating, which we thought was really valuable. And there’s currently no way in Cursor web to edit these files.

swyx: Yeah. Except you like open up the PR and go into GitHub and do the thing.

Jonas: Yeah.

swyx: Which is annoying.

Jonas: Just tell the agent,

swyx: I have criticized open AI for this. Because Open AI is Codex app doesn’t have a file editor, like it has file viewer, but isn’t a file editor.

Jonas: Do you use the file viewer a lot?

swyx: No. I understand, but like sometimes I want it, the one way to do it is like freaking going to no, they have a open in cursor button or open an antigravity or, opening whatever and people pointed that.

So I was, I was part of the early testers group people pointed that and they were like, this is like a design smell. It’s like you actually want a VS. Code fork that has all these things, but also a file editor. And they were like, no, just trust us.

Jonas: Yeah. I think we as Cursor will want to, as a product, offer the [00:34:00] whole spectrum and so you want to be able to.

Work at really high levels of abstraction and double click and see the lowest level. That’s important. But I also think that like you won’t be doing that in Slack. And so there are surfaces and ways of interacting where in some cases limiting the UX capabilities makes for a cleaner experience that’s more simple and drives people into these new patterns where even locally we kicked off joking about this.

People like don’t really edit files, hand code anymore. And so we want to build for where that’s going and not where it’s been

swyx: a lot of cool stuff. And Okay. I have a couple more.

Full Stack Hosting Debate

swyx: So observations about the design elements about these things. One of the things that I’m always thinking about is cursor and other peers of cursor start from like the Devrel tools and work their way towards cloud agents.

Other people, like the lovable and bolts of the world start with here’s like the vibe code. Full cloud thing. They were already cloud edges before anyone else cloud edges and we will give you the full deploy platform. So we own the whole loop. We own all the infrastructure, we own, we, we have the logs, we have the the live site, [00:35:00] whatever.

And you can do that cycle cursor doesn’t own that cycle even today. You don’t have the versal, you don’t have the, you whatever deploy infrastructure that, that you’re gonna have, which gives you powers because anyone can use it. And any enterprise who, whatever you infra, I don’t care. But then also gives you limitations as to how much you can actually fully debug end to end.

I guess I’m just putting out there that like is there a future where there’s like full stack cursor where like cursor apps.com where like I host my cursor site this, which is basically a verse clone, right? I don’t know.

Jonas: I think that’s a interesting question to be asking, and I think like the logic that you laid out for how you would get there is logic that I largely agree with.

swyx: Yeah. Yeah.

Jonas: I think right now we’re really focused on what we see as the next big bottleneck and because things like the Datadog MCP exist, yeah. I don’t think that the best way we can help our customers ship more software. Is by building a hosting solution right now,

swyx: by the way, these are things I’ve actually discussed with some of the companies I just named.

Jonas: Yeah, for sure. Right now, just this big bottleneck is getting the code out there and also [00:36:00] unlike a lovable in the bolt, we focus much more on existing software. And the zero to one greenfield is just a very different problem. Imagine going to a Shopify and convincing them to deploy on your deployment solution.

That’s very different and I think will take much longer to see how that works. May never happen relative to, oh, it’s like a zero to one app.

swyx: I’ll say. It’s tempting because look like 50% of your apps are versal, superb base tailwind react it’s the stack. It’s what everyone does.

So I it’s kinda interesting.

Jonas: Yeah.

Model Choice and Auto Routing

swyx: The other thing is the model select dying. Right now in cloud agents, it’s stuck down, bottom left. Sure it’s Codex High today, but do I care if it’s suddenly switched to Opus? Probably not.

Samantha: We definitely wanna give people a choice across models because I feel like it, the meta change is very frequently.

I was a big like Opus 4.5 Maximalist, and when codex 5.3 came out, I hard, hard switch. So that’s all I use now.

swyx: Yeah. Agreed. I don’t know if, but basically like when I use it in Slack, [00:37:00] right? Cursor does a very good job of exposing yeah. Cursors. If people go use it, here’s the model we’re using.

Yeah. Here’s how you switch if you want. But otherwise it’s like extracted away, which is like beautiful because then you actually, you should decide.

Jonas: Yeah, I think we want to be doing more with defaults.

swyx: Yeah.

Jonas: Where we can suggest things to people. A thing that we have in the editor, the desktop app is auto, which will route your request and do things there.

So I think we will want to do something like that for cloud agents as well. We haven’t done it yet. And so I think. We have both people like Sam, who are very savvy and want know exactly what model they want, and we also have people that want us to pick the best model for them because we have amazing people like Sam and we, we are the experts.

Yeah. We have both the traffic and the internal taste and experience to know what we think is best.

swyx: Yeah. I have this ongoing pieces of agent lab versus model lab. And to me, cursor and other companies are example of an agent lab that is, building a new playbook that is different from a model lab where it’s like very GP heavy Olo.

So obviously has a research [00:38:00] team. And my thesis is like you just, every agent lab is going to have a router because you’re going to be asked like, what’s what. I don’t keep up to every day. I’m not a Sam, I don’t keep up every day for using you as sample the arm arbitrator of taste. Put me on CRI Auto.

Is it free? It’s not free.

Jonas: Auto’s not free, but there’s different pricing tiers. Yeah.

swyx: Put me on Chris. You decide from me based on all the other people you know better than me. And I think every agent lab should basically end up doing this because that actually gives you extra power because you like people stop carrying or having loyalty with one lab.

Jonas: Yeah.

Best Of N and Model Councils

Jonas: Two other maybe interesting things that I don’t know how much they’re on your radar are one the best event thing we mentioned where running different models head to head is actually quite interesting because

swyx: which exists in cursor.

Jonas: That exists in cur ID and web. So the problem is where do you run them?

swyx: Okay.

Jonas: And so I, I can share my screen if that’s interesting. Yeahinteresting.

swyx: Yeah. Yeah. Obviously parallel agents, very popal.

Jonas: Yes, exactly. Parallel agents

swyx: in you mind. Are they the same thing? Best event and parallel agents? I don’t want to [00:39:00] put words in your mouth.

Jonas: Best event is a subset of parallel agents where they’re running on the same prompt.

That would be my answer. So this is what that looks like. And so here in this dropdown picker, I can just select multiple models.

swyx: Yeah.

Jonas: And now if I do a prompt, I’m going to do something silly. I am running these five models.

swyx: Okay. This is this fake clone, of course. The 2.0 yeah.

Jonas: Yes, exactly. But they’re running so the cursor 2.0, you can do desktop or cloud.

So this is cloud specifically where the benefit over work trees is that they have their own VMs and can run commands and won’t try to kill ports that the other one is running. Which are some of the pains. These are all

swyx: called work trees?

Jonas: No, these are all cloud agents with their own VMs.

swyx: Okay. But

Jonas: When you do it locally, sometimes people do work trees and that’s been the main way that people have set out parallel so far.

I’ve gotta say.

swyx: That’s so confusing for folks.

Jonas: Yeah.

swyx: No one knows what work trees are.

Jonas: Exactly. I think we’re phasing out work trees.

swyx: Really.

Jonas: Yeah.

swyx: Okay.

Samantha: But yeah. And one other thing I would say though on the multimodel choice, [00:40:00] so this is another experiment that we ran last year and the decide to ship at that time but may come back to, and there was an interesting learning that’s relevant for, these different model providers. It was something that would run a bunch of best of ends but then synthesize and basically run like a synthesizer layer of models. And that was other agents that would take LM Judge, but one that was also agentic and could write code. So it wasn’t just picking but also taking the learnings from two models or, and models that it was looking at and writing a new diff.

And what we found was that at the time at least, there were strengths to using models from different model providers as the base level of this process. Like basically you could get almost like a synergistic output that was better than having a very unified, like bottom model tier. So it was really interesting ‘cause it’s like potentially, even though even in the future when you have like maybe one model as ahead of the other for a little bit, there could be some benefit from having like multiple top tier models involved in like a [00:41:00] model swarm or whatever agent Swarm that you’re doing, that they each have strengths and weaknesses.

Yeah.

Jonas: Andre called this the council, right?

Samantha: Yeah, exactly. We actually, oh, that’s another internal command we have that Ian wrote slash council. Oh, and they some, yeah.

swyx: Yes. This idea is in various forms everywhere. And I think for me, like for me, the productization of it, you guys have done yeah, like this is very flexible, but.

If I were to add another Yeah, what your thing is on here it would be too much. I what, let’s say,

Samantha: Ideally it’s all, it’s something that the user can just choose and it all happens under the hood in a way where like you just get the benefit of that process at the end and better output basically, but don’t have to get too lost in the complexity of judging along the way.

Jonas: Okay.

Subagents for Context

Jonas: Another thing on the many agents, on different parallel agents that’s interesting is an idea that’s been around for a while as well that has started working recently is subagents. And so this is one other way to get agents of the different prompts and different goals and different models, [00:42:00] different vintages to work together.

Collaborate and delegate.

swyx: Yeah. I’m very like I like one of my, I always looking for this is the year of the blah, right? Yeah. I think one of the things on the blahs is subs. I think this is of but I haven’t used them in cursor. Are they fully formed or how do I honestly like an intro because do I form them from new every time?

Do I have fixed subagents? How are they different for slash commands? There’s all these like really basic questions that no one stops to answer for people because everyone’s just like too busy launching. We have to

Samantha: honestly, you could, you can see them in cursor now if you just say spin up like 50 subagents to, so cursor defines

swyx: what Subagents.

Yeah.

Samantha: Yeah. So basically I think I shouldn’t speak for the whole subagents team. This is like a different team that’s been working on this, but our thesis or thing that we saw internally is that like they’re great for context management for kind of long running threads, or if you’re trying to just throw more compute at something.

We have strongly used, almost like a generic task interface where then the main agent can define [00:43:00] like what goes into the subagent. So if I say explore my code base, it might decide to spin up an explore subagent and or might decide to spin up five explore subagent.

swyx: But I don’t get to set what those subagent are, right?

It’s all defined by a model.

Samantha: I think. I actually would have to refresh myself on the sub agent interface.

Jonas: There are some built-in ones like the explore subagent is free pre-built. But you can also instruct the model to use other subagents and then it will. And one other example of a built-in subagent is I actually just kicked one off in cursor and I can show you what that looks like.

swyx: Yes. Because I tried to do this in pure prompt space.

Jonas: So this is the desktop app? Yeah. Yeah. And that’s

swyx: all you need to do, right? Yeah.

Jonas: That’s all you need to do. So I said use a sub agent to explore and I think, yeah, so I can even click in and see what the subagent is working on here. It ran some fine command and this is a composer under the hood.

Even though my main model is Opus, it does smart routing to take, like in this instance the explorer sort of requires reading a ton of things. And so a faster model is really useful to get an [00:44:00] answer quickly, but that this is what subagent look like. And I think we wanted to do a lot more to expose hooks and ways for people to configure these.

Another example of a cus sort of builtin subagent is the computer use subagent in the cloud agents, where we found that those trajectories can be long and involve a lot of images obviously, and execution of some testing verification task. We wanted to use that models that are particularly good at that.

So that’s one reason to use subagents. And then the other reason to use subagents is we want contexts to be summarized reduced down at a subagent level. That’s a really neat boundary at which to compress that rollout and testing into a final message that agent writes that then gets passed into the parent rather than having to do some global compaction or something like that.

swyx: Awesome. Cool. While we’re in the subagents conversation, I can’t do a cursor conversation and not talk about listen stuff. What is that? What is what? He built a browser. He built an os. Yes. And he [00:45:00] experimented with a lot of different architectures and basically ended up reinventing the software engineer org chart.

This is all cool, but what’s your take? What’s, is there any hole behind the side? The scenes stories about that kind of, that whole adventure.

Samantha: Some of those experiments have found their way into a feature that’s available in cloud agents now, the long running agent mode internally, we call it grind mode.

And I think there’s like some hint of grind mode accessible in the picker today. ‘cause you can do choose grind until done. And so that was really the result of experiments that Wilson started in this vein where he I think the Ralph Wigga loop was like floating around at the time, but it was something he also independently found and he was experimenting with.

And that was what led to this product surface.

swyx: And it is just simple idea of have criteria for completion and do not. Until you complete,

Samantha: there’s a bit more complexity as well in, in our implementation. Like there’s a specific, you have to start out by aligning and there’s like a planning stage where it will work with you and it will not get like start grind execution mode until it’s decided that the [00:46:00] plan is amenable to both of you.

Basically,

swyx: I refuse to work until you make me happy.

Jonas: We found that it’s really important where people would give like very underspecified prompt and then expect it to come back with magic. And if it’s gonna go off and work for three minutes, that’s one thing. When it’s gonna go off and work for three days, probably should spend like a few hours upfront making sure that you have communicated what you actually want.

swyx: Yeah. And just to like really drive from the point. We really mean three days that No, no

Jonas: human. Oh yeah. We’ve had three day months innovation whatsoever.

Samantha: I don’t know what the record is, but there’s been a long time with the grants

Jonas: and so the thing that is available in cursor. The long running agent is if you wanna think about it, very abstractly that is like one worker node.

Whereas what built the browser is a society of workers and planners and different agents collaborating. Because we started building the browser with one worker node at the time, that was just the agent. And it became one worker node when we realized that the throughput of the system was not where it needed to be [00:47:00] to get something as large of a scale as the browser done.

swyx: Yeah.

Jonas: And so this has also become a really big mental model for us with cloud, cloud agents is there’s the classic engineering latency throughput trade-offs. And so you know, the code is water flowing through a pipe. The, we think that over the coming months, the big unlock is not going to be one person with a model getting more done, like the water flowing faster and we’ll be making the pipe much wider and so ing more, whether that’s swarms of agents or parallel agents, both of those are things that contribute to getting.

Much more done in the same amount of time, but any one of those tasks doesn’t necessarily need to get done that quickly. And throughput is this really big thing where if you see the system of a hundred concurrent agents outputting thousands of tokens a second, you can’t go back like that.

Just you see a glimpse of the future where obviously there are many caveats. Like no one is using this browser. IRL. There’s like a bunch of things not quite right yet, but we are going to get to systems that produce real production [00:48:00] code at the scale much sooner than people think. And it forces you to think what even happens to production systems. Like we’ve broken our GitHub actions recently because we have so many agents like producing and pushing code that like CICD is just overloaded. ‘cause suddenly it’s like effectively weg grew, cursor’s growing very quickly anyway, but you grow head count, 10 x when people run 10 x as many agents.

And so a lot of these systems, exactly, a lot of these systems will need to adapt.

swyx: It also reminds me, we, we all, the three of us live in the app layer, but if you talk to the researchers who are doing RL infrastructure, it’s the same thing. It’s like all these parallel rollouts and scheduling them and making sure as much throughput as possible goes through them.

Yeah, it’s the same thing.

Jonas: We were talking briefly before we started recording. You were mentioning memory chips and some of the shortages there. The other thing that I think is just like hard to wrap your head around the scale of the system that was building the browser, the concurrency there.

If Sam and I both have a system like that running for us, [00:49:00] shipping our software. The amount of inference that we’re going to need per developer is just really mind-boggling. And that makes, sometimes when I think about that, I think that even with, the most optimistic projections for what we’re going to need in terms of buildout, our underestimating, the extent to which these swarm systems can like churn at scale to produce code that is valuable to the economy.

And,

swyx: yeah, you can cut this if it’s sensitive, but I was just Do you have estimates of how much your token consumption is?

Jonas: Like per developer?

swyx: Yeah. Or yourself. I don’t need like comfy average. I just curious. I

Samantha: feel like I, for a while I wasn’t an admin on the usage dashboard, so I like wasn’t able to actually see, but it was a,

swyx: mine has gone up.

Samantha: Oh yeah.

swyx: But I think

Samantha: it’s in terms of how much work I’m doing, it’s more like I have no worries about developers losing their jobs, at least in the near term. ‘cause I feel like that’s a more broad discussion.

swyx: Yeah. Yeah. You went there. I didn’t go, I wasn’t going there.

I was just like how much more are you using?

Samantha: There’s so much stuff to be built. And so I feel like I’m basically just [00:50:00] trying to constantly I have more ambitions than I did before. Yes. Personally. Yes. So can’t speak to the broader thing. But for me it’s like I’m busier than ever before.

I’m using more tokens and I am also doing more things.

Jonas: Yeah. Yeah. I don’t have the stats for myself, but I think broadly a thing that we’ve seen, that we expect to continue is J’S paradox. Where

swyx: you can’t do it in our podcast without seeing

Jonas: it. Exactly. We’ve done it. Now we can wrap. We’ve done, we said the words.

Phase one tab auto complete people paid like 20 bucks a month. And that was great. Phase two where you were iterating with these local models. Today people pay like hundreds of dollars a month. I think as we think about these highly parallel kind of agents running off for a long times in their own VM system, we are already at that point where people will be spending thousands of dollars a month per human, and I think potentially tens of thousands and beyond, where it’s not like we are greedy for like capturing more money, but what happens is just individuals get that much more leverage.

And if one person can do as much as 10 people, yeah. That tool that allows ‘em to do that is going to be tremendously valuable [00:51:00] and worth investing in and taking the best thing that exists.

swyx: One more question on just the cursor in general and then open-ended for you guys to plug whatever you wanna put.

How is Cursor hiring these days?

Samantha: What do you mean by how?

swyx: So obviously lead code is dead. Oh,

Samantha: okay.

swyx: Everyone says work trial. Different people have different levels of adoption of agents. Some people can really adopt can be much more productive. But other people, you just need to give them a little bit of time.

And sometimes they’ve never lived in a token rich place like cursor.

And once you live in a token rich place, you’re you just work differently. But you need to have done that. And a lot of people anyway, it was just open-ended. Like how has agentic engineering, agentic coding changed your opinions on hiring?

Is there any like broad like insights? Yeah.

Jonas: Basically I’m asking this for other people, right? Yeah, totally. Totally. To hear Sam’s opinion, we haven’t talked about this the two of us. I think that we don’t see necessarily being great at the latest thing with AI coding as a prerequisite.

I do think that’s a sign that people are keeping up and [00:52:00] curious and willing to upscale themselves in what’s happening because. As we were talking about the last three months, the game has completely changed. It’s like what I do all day is very different.

swyx: Like it’s my job and I can’t,

Jonas: Yeah, totally.

I do think that still as Sam was saying, the fundamentals remain important in the current age and being able to go and double click down. And models today do still have weaknesses where if you let them run for too long without cleaning up and refactoring, the coke will get sloppy and there’ll be bad abstractions.

And so you still do need humans that like have built systems before, no good patterns when they see them and know where to steer things.

Samantha: I would agree with that. I would say again, cursor also operates very quickly and leveraging ag agentic engineering is probably one reason why that’s possible in this current moment.

I think in the past it was just like people coding quickly and now there’s like people who use agents to move faster as well. So it’s part of our process will always look for we’ll select for kind of that ability to make good decisions quickly and move well in this environment.

And so I think being able to [00:53:00] figure out how to use agents to help you do that is an important part of it too.

swyx: Yeah. Okay. The fork in the road, either predictions for the end of the year, if you have any, or PUDs.

Jonas: Evictions are not going to go well.

Samantha: I know it’s hard.

swyx: They’re so hard. Get it wrong.

It’s okay. Just, yeah.

Jonas: One other plug that may be interesting that I feel like we touched on but haven’t talked a ton about is a thing that the kind of these new interfaces and this parallelism enables is the ability to hop back and forth between threads really quickly. And so a thing that we have,

swyx: you wanna show something or,

Jonas: yeah, I can show something.

A thing that we have felt with local agents is this pain around contact switching. And you have one agent that went off and did some work and another agent that, that did something else. And so here by having, I just have three tabs open, let’s say, but I can very quickly, hop in here.

This is an example I showed earlier, but the actual workflow here I think is really different in a way that may not be obvious, where, I start the morning, I kick off 10 agents or something, the first one of them [00:54:00] finishes, come in, watch the video either as close. And so I might send a follow up.

I might say, Hey, make it red, or I might hop into the desktop and try it out. And within, 90, 120 seconds, I’ve kicked this one back off. And either started the merge process like CI is running now and I’ll come back to it later or it’s off with some additional follow up information. And then I can hop into the next one.

And then the next one I hop in and I’m like, okay, this looks interesting. Actually try it out for real in the app. I want to see it in action, not just in the gallery. So I can kick that off and the agent will go and work on that because maybe I wanted to try it out, like what the button looks like in the actual thing.

And then here I might hop in as well and, check the video here or do something. And so you’re really parallelizing much more and follow up here, check in there. It’s much more this higher level of abstraction and having the different desktops where you can hop back and forth and you’re [00:55:00] not like, oh, I checked out this branch.

Oh, where was that work tree again? Yeah. It’s really like solving for that which we’ve ourselves have struggled with in cursor and these local agents to be like, where was that diff again? It’s lost in some work tree. Never gonna find it. Oh, my local thing is rebuilding. Oh, just make another one.

That, that’s what you end up with and then you wait for five more minutes for it to run. And so this is really like a new way of just paralleling that we found to be really fun, honestly. Yeah. Where you’re just hopping in and injecting taste and you’re like that doesn’t quite feel right.

Oh, actually this is not architected quite right, but you’re just focusing on those like taste interesting questions.

Samantha: For me, the cloud ecosystem too also enabled this to be like, something that is like adding productivity to my dead time, like commuting or like overnight or something like that.

The fact that I don’t have to leave my computer open,

swyx: there’s no cursor, there is a cursor mobile app.

Samantha: If there is, I’m not sure. It’s like the current thing. We, I use it on my phone all the time, just on the web. So pretty good experience there for checking [00:56:00] in. Yeah. And un unlocking. I think, yeah. You can see the videos and stuff in the web app, which is awesome.

Yeah.

Jonas: Yeah.

swyx: I think this is one that the a DD one inherited the earth, like the, if you’re like, your attention span is cooked, but you still can manage, like actually this is good for you. Yeah. But also I think this is where the coding tools start coming into conflict with the productivity tools where like the linear the canman boards, because what you have there is cool, but you know what, you actually need a cabin board. Which people have vibe, vibe, cam, van is out there. Open source. I’m sure you guys have talked about it, but we’ll start to conflict because actually the code doesn’t matter anymore.

It’s the process of the human interacting and checking in. And seeing, like getting the world of warcrafts sound package to go like work or whatever. It’s like job done or, I don’t know. It’s like an interesting like future productivity thing.

Samantha: Yeah.

swyx: I also think like another big theme like last year li is called like the, your coding agents.

This year another like coding agents spill over to the real world into cloud cowork and all the other stuff. Yeah. I’m sure cursor’s gonna focus on software, but let’s call it like open claw is like extremely [00:57:00] mind expanding in terms of I did not know that could happen.

Jonas: Yeah.

swyx: And it’s all based on a coding agent based totally.

Jonas: And I think one of the things that like talking to, friends and family that are not in the software world that’s interesting is I do. Speeding up predictions. I do think that we are going to start seeing other industries go through what software development has started going through.

I think by virtue of how good models are at writing software and how early adopter the people building the new technology are and trying it out and applying it to themselves, that’s certain kinds of shifts will happen too to other industries. And there’s a lot to be learned from how that’s gone down and is continuing to go down in software.

In terms of, all the interesting questions about to what point do people get more leverage, when do you start changing the role to become much more generalist? Like, all of these questions that we’ve seen some data on, but we’ll see a lot more in the coming months. That will happen everywhere.

swyx: Sammy party thoughts? Any flus of your own?

Samantha: Not really. [00:58:00] It’s fine. I feel we covered so much good draft. We covered it. We covered a lot. Coming up with a prediction. I just think agents are gonna keep getting better. Gonna stop doing as much manual coding, probably zero lines of code written in the whole month of December this year by myself.

A hundred percent agents as a personal prediction, but

swyx: oh, you’re not as zero today.

What in what cases?

Samantha: I think honestly, it’s 1% if I like, just am like, get frustrated and I’m like, I don’t wanna go have it tell an agent to change this one thing. But

Jonas: prompting sometimes I feel like working on prompts sometimes.

Yeah. I still go in and manually edit because it’s so like bare intent transfer that like telling the agent what I want. It’s like writing an essay where I don’t use agents to write essays yet because the process of writing it is the thinking.

Samantha: I still can’t stand AI generated writing. So yeah, I can also can’t have the agent write prompts.

swyx: So no D Spy, no jpa, nothing like that here.

Jonas: We have some internal tooling around some of the prompt optimization things, but there’s a fair amount of just what concepts do I need to communicate to the agent or the model.

swyx: I also noticed another thing I’m also [00:59:00] looking for is voice.

I noticed that you didn’t use your voice to code even open ai. When we do podcasts with them, they don’t use their voice. Yeah. And I’m like at some point this gets good. You can stop typing.

Samantha: We have some people who like that a lot internally, and I think we’ll be experimenting in that space too, for sure.

Jonas: Do you use voice log?

swyx: Not a lot. Sometimes that’s bound to my caps log, so I can press it. I just,

Jonas: and when you use it, do you want it to talk back or you just want

swyx: Yeah,

Jonas: just dump in. Yeah. Yeah.

swyx: But like the brain dump is good. Yeah. Because you can interrupt yourself. You can go on a tangent, whatever.

It just captures everything. Yeah. And lop it into all m, it’s fine.

Jonas: Yeah. The way that we did this with Auto Tab was people would record full screen recordings with audio to teach the model, like how to do a task. And one of the funny things that we learned was people would use their Siri voice, where they would start talking in like short, stilted sentences and enunciate really clearly because they were used to, they last used AI two years ago where you had to

swyx: apple has damaged like an entire generation of people’s expectations.

Jonas: Exactly. And we had to be like, no you’re very native, so [01:00:00] you do this, but just dump everything in. You can say you can repeat yourself. You can contradict yourself. The models are smart enough to figure it out,

swyx: but it’s still very bad. So voice coding was always, I considered like the hardest part because you have to say like technical things that pel like spelling matters, capitalization matters and like it’s all not in voice.

So we’ll see. So far it’s been more sort of emotional companionship, that kind of stuff, but at some point it’s gonna hit voice coding.

Jonas: Yeah. I have a prediction for you. I predict that by the end of the year, the volume on, I think it will take longer than people think and longer than we think for cloud and agents working in their own boxes to surpass local agents.

But I think that crossover will happen before the end of the year and probably by the end of the year, agents running in the cloud will be a multi, like more than two x the volume of local agents.

swyx: Okay. You’re leaving me an opening. What’s not good today?

Jonas: Yeah, there’s a bunch of hard things. So one of them is just getting those [01:01:00] sandboxes to be really good and a thing that was part of this launch that we spend in inordinate amount of time on is cursor.com/onboard where you pick a repo, add secrets, give it access to things, and the agent just goes off and installs things.

swyx: Yes, I think all the whole thing. That was my favorite.

Jonas: Yeah, we worked a lot on that. Sam and I in particular spent a lot of late nights making that good, but there’s still a lot to do there, right? Set up 1, 2, 2 things. Maybe it’s too slow. It’s too slow. Working on it set up is not like a unitary thing where everything is set up or not, right?

Like things will break over time. You have new dependencies, you need access to new systems, like you change where your database lives. So that’s one part of it. And then the other part of it is, having these agents run in the cloud and be more autonomous. We’ve really started to see the lack of memory.

And Sam, as someone who’s thought a lot about this once you start getting the model kind of doing, operating the code base, there’s more particularities that are not it’s not just a read file tool. It needs to know how do I start up the backend, how do I check the status [01:02:00] of the backend?

That’s very particular to your code base. And even if it’s great at NPM Run watch or whatever the default things are, there’s always quirks. Everyone has quirks. And getting the model good at those things will require more work. And we’re working on that. But we think that will be one of the big unlocks, is having them be onboarded not only in terms of their environment, but also in terms of their understanding of design trade-offs, how the code base works, how to be a good developer in any one code base.

swyx: It’s lot crier rules. It’s gonna be something else. Is it gonna be a file? Is it. We just call in either markdown file a different name, and

Samantha: I don’t know. One thing that we learned at, could we be in cursor of the company this year? There, there’s a really great blog post that the Judi and the other people in the agent quality team put out about dynamic file context.

swyx: Is that your team is the different team?

Samantha: Different team, yeah. And they were working on basically doing a lot everything, file system, everything is file system. And so a lot of my thinking personally on memory this past year has changed to be more aligned with that, where it’s like giving the agent pointers to things, annotations [01:03:00] to things.

The second thing I think that I’ve started to think differently about memory is a subset of agent self-audit ability and self-awareness. Basically like the agent might wanna propose annotations or links or memory like files to itself when it finds that there’s like some gap in its functionality in its own harness that might need to be filled by like some piece of information on a semi-permanent basis.

But there’s a whole bunch of other things that are a side effect of self auditability that are really interesting, like potentially finding like conflicting instructions or like skills and rules that like might be like, eh, these are bugging each other. And also things like fixing like Devrel X problems that it runs into.

I think that basically the dynamic file system stuff is probably very promising from memory. And there’s also this notion of needing to have the agent be a little bit more self-aware in terms of being able to identify gaps in its own functionality and decide how to fill them.

Jonas: That’s such a good point.

Like self-awareness broadly has been a really big thing that I think Sam has pushed us to [01:04:00] do more and more of where the agent should understand how its environment works, it should understand how secrets work. Like it needs to be self-aware about its own harness and its environment. And then, and you

swyx: think this is not inherent in the model you have to do.

Jonas: It’s specifics, right? If it’s running in cursor versus some other sandbox that’s a bit different. And then the other part of it that starts to get really interesting is when the model starts editing its own system. Prompt.

swyx: Yeah.

Jonas: What does that even mean? How do you do that safely and then way over

swyx: do that?

This is just research, right? This isn’t, this is

Jonas: I think it will do that. Yeah. It will manage its own context. And so system prompt is part of the context, and you can argue about

Samantha: Yeah, like other things that it might decide to turn off or on depending, and all those, self-awareness to us in this context is not like the model itself, having a notion of consciousness, but more like knowing like what system it’s operating in and the constraints of that system and potentially being able to have agency in optimizing itself to operate best in the, in that system.

This was like one of the [01:05:00] first things I learned at DOT when we launched was that I we had made the model or made the agent or. Whatever we would call it. At that time, it was far less, agentic made the product work very well at a certain number of things, but didn’t have complete self-awareness of like its own boundaries.

So people would be like, Hey, can you do this thing? And the thing was there and could be done and the and the product would be like, oh no. And I’d be like, but you can. And so like basically like that was one of the earliest things I found is

swyx: believe in yourself.

Samantha: I know as a product developer, like it needs to both be able to do the thing and it needs to have complete knowledge of its ability to do the thing.

Those are not always obviously the same like part of the prompt at all.

Yeah.

swyx: Yeah.

Samantha: It’s something that I think has continued to be a theme in the ecosystem that users will often attribute increased intelligence to a system that is more highly self-aware and is more able to like, manipulate itself to do well in a system.

If that makes sense.

swyx: Yeah. This is more abstract than I ever thought would get at Thisor discussion. Cool. That isn’t the kind of [01:06:00] conversation that you have

Samantha: in, we talk about this stuff all the time to

swyx: improving

Samantha: Yeah.

swyx: Agents in general.

Jonas: Yeah. I think to your point right about the agent layer and thinking a lot about models and the harness and the product and the affordances like that.

Yeah. Falls from the

swyx: No, I mean you guys are like my sort of needing example what an agent lab looks like and can be successful and I think people always hungry for insights into how you guys operate, so thank you for taking the time to share.

Samantha: Yeah. Thanks for coming.

Yeah. Thank you.

The reception to our recent post on Code Reviews has been strong. Catch up!

Amid a maelstrom of discussion on whether or not AI is killing SaaS, one of the top publicly listed SaaS companies in the world has just reported record revenues, clearing well over $1.1B in ARR for the first time with a 28% margin. As we comment on the pod, Aaron Levie is the rare public company CEO equally at home in both worlds of Silicon Valley and Wall Street/Main Street, by day helping 70% of the Fortune 500 with their Enterprise Advanced Suite, and yet by night is often found in the basements of early startups and tweeting viral insights about the future of agents.

Now that both Cursor, Cloudflare, Perplexity, Anthropic and more have made Filesystems and Sandboxes and various forms of “Just Give the Agent a Box” cool (not just cool; it is now one of the single hottest areas in AI infrastructure growing 100% MoM), we find it a delightfully appropriate time to do the episode with the OG CEO who has been giving humans and computers Boxes since he was a college dropout pitching VCs at a Michael Arrington house party.

Enjoy our special pod, with fan favorite returning guest/guest cohost Jeff Huber!

Note: We didn’t directly discuss the AI vs SaaS debate - Aaron has done many, many, many other podcasts on that, and you should read his definitive essay on it. Most commentators do not understand SaaS businesses because they have never scaled one themselves, and deeply reflected on what the true value proposition of SaaS is.

We also discuss Your Company is a Filesystem:

We also shoutout CTO Ben Kus’ and the AI team, who talked about the technical architecture and will return for AIE WF 2026.

Full Video Episode

Timestamps

* 00:00 Adapting Work for Agents

* 01:29 Why Every Agent Needs a Box

* 04:38 Agent Governance and Identity

* 11:28 Why Coding Agents Took Off First

* 21:42 Context Engineering and Search Limits

* 31:29 Inside Agent Evals

* 33:23 Industries and Datasets

* 35:22 Building the Agent Team

* 38:50 Read Write Agent Workflows

* 41:54 Docs Graphs and Founder Mode

* 55:38 Token FOMO Culture

* 56:31 Production Function Secrets

* 01:01:08 Film Roots to Box

* 01:03:38 AI Future of Movies

* 01:06:47 Media DevRel and Engineering

Transcript

Adapting Work for Agents

Aaron Levie: Like you don’t write code, you talk to an agent and it goes and does it for you, and you may be at best review it. That’s even probably like, like largely not even what you’re doing. What’s happening is we are changing our work to make the agents effective. In that model, the agent didn’t really adapt to how we work.

We basically adapted to how the agent works. All of the economy has to go through that exact same evolution. Right now, it’s a huge asset and an advantage for the teams that do it early and that are kinda wired into doing this ‘cause you’ll see compounding returns. But that’s just gonna take a while for most companies to actually go and get this deployed.

swyx: Welcome to the Lane Space Pod. We’re back in the chroma studio with uh, chroma, CEO, Jeff Hoover. Welcome returning guest now guest host.

Aaron Levie: It’s a pleasure. Wow. How’d you get upgraded to, uh, to that?

swyx: Because he’s like the perfect guy to be guest those for you.

Aaron Levie: That makes sense actually, for We love context. We, we both really love context le we really do.

We really do.

swyx: Uh, and we’re here with, uh, Aaron Levy. Welcome.

Aaron Levie: Thank you. Good to, uh, good to be [00:01:00] here.

swyx: Uh, yeah. So we’ve all met offline and like chatted a little bit, but like, it’s always nice to get these things in person and conversation. Yeah. You just started off with so much energy. You’re, you’re super excited about agents.

I love

Aaron Levie: agents.

swyx: Yeah. Open claw. Just got by, got bought by OpenAI. No, not bought, but you know, you know what I mean?

Aaron Levie: Some, some, you know, acquihire. Executive

swyx: hire.

Aaron Levie: Executive hire. Okay. Executive hire. Say,

swyx: hey, that’s my term. Okay. Um, what are you pounding the table on on agents? You have so many insightful tweets.

Why Every Agent Needs a Box

Aaron Levie: Well, the thing that, that we get super excited by that I think is probably, you know, should be relatively obvious is we’ve, we’ve built a platform to help enterprises manage their files and their, their corporate files and the permissions of who has access to those files and the sharing collaboration of those files.

All of those files contain really, really important information for the enterprise. It might have your contracts, it might have your research materials, it might have marketing information, it might have your memos. All that data obviously has, you know, predominantly been used by humans. [00:02:00] But there’s been one really interesting problem, which is that, you know, humans only really work with their files during an active engagement with them, and they kind of go away and you don’t really see them for a long time.

And all of a sudden, uh, with the power of AI and AI agents, all of that data becomes extremely relevant as this ongoing source of, of answers to new questions of data that will transform into, into something else that, that produces value in your organization. It, it contains the answer to the new employee that’s onboarding, that needs to ramp up on a project.

Um, it contains the answer to the right thing to sell a customer when you’re having a conversation to them, with them contains the roadmap information that’s gonna produce the next feature. So all that data. That previously we’ve been just sort of storing and, and you know, occasionally forgetting about, ‘cause we’re only working on the new active stuff.

All of that information becomes valuable to the enterprise and it’s gonna become extremely valuable to end users because now they can have agents go find what they’re looking for and produce new, new [00:03:00] value and new data on that information. And it’s gonna become incredibly valuable to agents because agents can roam around and do a bunch of work and they’re gonna need access to that data as well.

And um, and you know, sometimes that will be an agent that is sort of working on behalf of, of, of you and, and effectively as you as and, and they are kind of accessing all of the same information that you have access to and, and operating as you in the system. And then sometimes there’s gonna be agents that are just.

Effectively autonomous and kind of run on their own and, and you’re gonna collaborate and work with them kind of like you did another person. Open Claw being the most recent and maybe first real sort of, you know, kind of, you know, up updating everybody’s, you know, views of this landscape version of, of what that could look like, which is, okay, I have an agent.

It’s on its own system, it’s on its own computer, it has access to its own tools. I probably don’t give it access to my entire life. I probably communicate with it like I would an assistant or a colleague and then it, it sort of has this sandbox environment. So all of that has massive implications for a platform that manage that [00:04:00] enterprise data.

We think it’s gonna just transform how we work with all of the enterprise content that we work with, and we just have to make sure we’re building the right platform to support that.

swyx: The sort of shorthand I put it is as people build agents, everybody’s just realizing that every agent needs a box. Yes.

And it’s nice to be called box and just give everyone a box.

Aaron Levie: Hey, I if I, you know, if we can make that go viral, uh, like I, I think that that terminology, I, that’s the

swyx: tagline. Every agent

Aaron Levie: needs a box. Every agent needs a box. If we can make that the headline of this, I’m fine with this. And that’s the billboard I wanna like Yeah, exactly.

Every agent needs a box. Um, I like it. Can we ship this? Like,

swyx: okay, let’s do it. Yeah.

Aaron Levie: Uh, my work here is done and I got the value I needed outta this podcast Drinks.

swyx: Yeah.

Agent Governance and Identity

Aaron Levie: But, but, um, but, but, you know, so the thing that we, we kind of think about is, um, is, you know, whether you think the number 10 x or a hundred x or whatever the number is, we’re gonna have some order of magnitude more agents than people.

That’s inevitable. It has to happen. So then the question is, what is the infrastructure that’s needed to make all those agents effective in the enterprise? Make sure that they are well governed. Make sure they’re only doing [00:05:00] safe things on your information. Make sure that they’re not getting exposed. The data that they shouldn’t have access to.

There’s gonna be just incredibly spectacularly crazy security incidents that will happen with agents because you’ll prompt, inject an agent and sort of find your way through the CRM system and pull out data that you shouldn’t have access to. Oh, we

Jeff Huber: have God,

Aaron Levie: right? I mean, that’s just gonna happen all over the place, right?

So, so then the thing is, is how do you make sure you have the right security, the permissions, the access controls, the data governance. Um, we actually don’t yet exactly know in many cases how we’re gonna regulate some of these agents, right? If you think about an agent in financial services, does it have the exact same financial sort of, uh, requirements that a human did?

Or is it, is the risk fully on the human that was interacting or created the agent? All open questions, but no matter what, there’s gonna need to be a layer that manages the, the data they have access to, the workflows that they’re involved in, pulling up data from multiple systems. This is the new infrastructure opportunity in the era of agents.

swyx: You have a piece on agent identities, [00:06:00] which I think was today, um, which I think a lot of breaking news, the security, security people are talking about, right? Like you basically, I, I always think of this as like, well you need the human you and then there you need the agent. You

Aaron Levie: Yes.

swyx: And uh, well, I don’t know if it’s that simple, but is box going to have an opinion on that or you’re just gonna be like, well we’re just the sort of the, the source layer.

Yeah. Let’s Okta of zero handle that.

Aaron Levie: I think we’re gonna have an opinion and we will work with generally wherever the contours of the market end up. Um, and the reason that we’re gonna have an opinion more than other topics probably is because one of the biggest use cases for why your agent might need it, an identity is for file system access.

So thus we have to kind of think about this pretty deeply. And I think, uh, unless you’re like in our world thinking about this particular problem all day long, it might be, you know, like, why is this such a big deal? And the reason why it’s a really big deal is because sometimes sort of say, well just give the agent an, an account on the system and it just treats, treat it like every other type of user on the system.

The [00:07:00] problem is, is that I as Aaron don’t really have any responsibility over anybody else’s box account in our organization. I can’t see the box account of any other employee that I work with. I am not liable for anything that they do. And they have, I have, I have, you know, strict privacy requirements on everything that they’re able to, you know, that, that, that they work on.

Agents don’t have that, you know, don’t have those properties. The person who creates the agent probably is gonna, for the foreseeable future, take on a lot of the liability of what that agent does. That agent doesn’t deserve any privacy because, because it’s, you know, it can’t fully be autonomously operated and it doesn’t have any legal, you know, kind of, you know, responsibility.

So thus you can’t just be like, oh, well I’ll just create a bunch of accounts and then I’ll, I’ll kind of work with that agent and I’ll talk to it occasionally. Like you need oversight of that. And so then the question is, how do you have a world where the agent, sometimes you have oversight of, but what if that agent goes and works with other people?

That person over there is collaborating with the agent on something you shouldn’t have [00:08:00] access to what they’re doing. So we have all of these new boundaries that we’re gonna have to figure out of, of, you know, it’s really, really easy. So far we’ve been in, in easy mode. We’ve hit the easy button with ai, which is the agent just is you.

And when you’re in quad code and you’re in cursor, and you’re in Codex, you’re just, the agent is you. You’re offing into your services. It can do everything you can do. That’s the easy mode. The hard mode is agents are kind of running on their own. People check in with them occasionally, they’re doing things autonomously.

How do you give them access to resources in the enterprise and not dramatically increased the security risk and the risk that you might expose the wrong thing to somebody. These are all the new problems that we have to get solved. I like the identity layer and, and identity vendors as being a solution to that, but we’ll, we’ll need some opinions as well because so many of the use cases are these collaborative file system use cases, which is how do I give it an agent, a subset of my data?

Give it its own workspace as well. ‘cause it’s gonna need to store off its own information that would be relevant for it. And how do I have the right oversight into that? [00:09:00]

Jeff Huber: One thing, which, um, I think is kind interesting, think about is that you know, how humans work, right? Like I may not also just like give you access to the whole file.

I might like sit next to you and like scroll to this like one part of the file and just show you that like one part and like, you know,

swyx: partial file access.

Jeff Huber: I’m just saying I think like our, like RA does seem to be dead, right? Like you wanna say something is dead uhhuh probably RA is dead. And uh, like the auth story to me seems like incredibly unsolved and unaddressed by like the existing state of like AI vendors.

But

Aaron Levie: yeah, I think, um, we’re, I mean you’re taking obviously really to level limit that we probably need to solve for. Yeah. And we built an access control system that was, was kind of like, you know, its own little world for, for a long time. And um, and the idea was this, it’s a many to many collaboration system where I can give you any part of the file system.

And it’s a waterfall model. So if I give you higher up in the, in the, in the system, you get everything below. And that, that kind of created immense flexibility because I can kind of point you to any layer in the, in the tree, but then you’re gonna get access to everything kind of below it. And that [00:10:00] mostly is, is working in this, in this world.

But you do have to manage this issue, which is how do I create an agent that has access to some of my stuff and somebody else’s stuff as well. Mm-hmm. And which parts do I get to look at as the creator of the agent? And, and these are just brand new problems? Yeah. Crazy. And humans, when there was a human there that was really easy to do.

Like, like if the three of us were all sharing, there’d be a Venn diagram where we’d have an overlapping set of things we’ve shared, but then we’d have our own ways that we shared with each other. In an agent world, somebody needs to take responsibility for what that agent has access to and what they’re working on.

These are like the, some of the most probably, you know, boring problems for 98% of people on, on the internet, but they will be the problems that are the difference between can you actually have autonomous agents in an enterprise context

swyx: Yeah.

Aaron Levie: That are not leaking your data constantly.

swyx: No. Like, I mean, you know, I run a very, very small company for my conference and like we already have data sensitivity issues.

Yes. And some of my team members cannot see Yes. Uh, the others and like, I can’t imagine what it’s like to run a Fortune 500 and like, you have to [00:11:00] worry about this. I’m just kinda curious, like you, you talked to a lot like, like 70, 80% of your cus uh, of the Fortune 500, your customers.

Aaron Levie: Yep. 67%. Just so we’re being very

SE

swyx: precise.

So Yeah. I’m not

Aaron Levie: Okay. Okay.

swyx: Something I’m rounding up. Yes. Round up. I’m projecting to, for

Aaron Levie: the government.

swyx: I’m projecting to the end of the year.

Aaron Levie: Okay.

swyx: There you go.

Aaron Levie: You do make it sound like, like we, we, well we’ve gotta be on this. Like we’re, we’re taking way too long to get to 80%. Well,

swyx: no, I mean, so like. How are they approaching it?

Right? Because you’re, you don’t have a, you don’t have a final answer yet.

Why Coding Agents Took Off First

Aaron Levie: Well, okay, so, so this is actually, this is the stark reality that like, unfortunately is the kinda like pouring the water on the party a little bit.

swyx: Yes.

Aaron Levie: We all in Silicon Valley are like, have the absolute best conditions possible for AI ever.

And I think we all saw the dke, you know, kind of Dario podcast and this idea of AI coding. Why is that taken off? And, and we’re not yet fully seeing it everywhere else. Well, look, if you just like enumerated the list of properties that AI coding has and then compared it to other [00:12:00] knowledge work, let’s just, let’s just go through a few of them.

Generally speaking, you bring on a new engineer, they have access to a large swath of the code base. Like, there’s like very, like you, just, like new engineer comes on, they can just go and find the, the, the stuff that they, they need to work with. It’s a fully text in text out. Medium. It’s only, it’s just gonna be text at the end of the day.

So it’s like really great from a, from just a, uh, you know, kinda what the agent can work with. Obviously the models are super trained on that dataset. The labs themselves have a really strong, kind of self-reinforcing positive flywheel of why they need to do, you know, agent coding deeply. So then you get just better tooling, better services.

The actual developers of the AI are daily users of the, of the thing that they’re we’re working on versus like the, you know, probably there’s only like seven Claude Cowork legal plugin users at Anthropic any given day, but there’s like a couple thousand Claude code and you know, users every single day.

So just like, think about which one are they getting more feedback on. All day long. So you just go through this list. You have a, you know, everybody who’s a [00:13:00] developer by definition is technical so they can go install the latest thing. We’re all generally online, or at least, you know, kinda the weird ones are, and we’re all talking to each other, sharing best practices, like that’s like already eight differences.

Versus the rest of the economy. Every other part of the economy has like, like six to seven headwinds relative to that list. You go into a company, you’re a banker in financial services, you have access to like a, a tiny little subset of the total data that’s gonna be relevant to do your job. And you’re have to start to go and talk to a bunch of people to get the right data to do your job because Sally didn’t add you to that deal room, you know, folder.

And that that, you know, the information is actually in a completely different organization that you now have to go in and, and sort of run into. And it’s like you have this endless list of access controls and security. As, as you talked about, you have a medium, which is not, it’s not just text, right? You have, you have a zoom call that, that you’re getting all of the requirements from the customer.

You have a lot of in-person conversations and you’re doing in-person sales and like how do you ever [00:14:00] digitize all of that information? Um, you know, I think a lot of people got upset with this idea that the code base has all the context, um, that I don’t know if you follow, you know, did you follow some of that conversation that that went viral?

Is like, you know, it’s not that simple that, that the code base doesn’t have all the knowledge, but like it’s a lot, you’re a lot better off than you are with other areas of knowledge work. Like you, we like, we like have documentation practices, you write specifications. Those things don’t exist for like 80% of work that happens in the enterprise.

That’s the divide that we have, which is, which is AI coding has, has just fully, you know, where we’ve reached escape velocity of how powerful this stuff is, and then we’re gonna have to find a way to bring that same energy and momentum, but to all these other areas of knowledge work. Where the tools aren’t there, the data’s not set up to be there.

The access controls don’t make it that easy. The context engineering is an incredibly hard problem because again, you have access control challenges, you have different data formats. You have end users that are gonna need to kind of be kind of trained through this as opposed to their adopting [00:15:00] these tools in their free time.

That’s where the Fortune 500 is. And so we, I think, you know, have to be prepared as an industry where we are gonna be on a multi-year march to, to be able to bring agents to the enterprise for these workflows. And I think probably the, the thing that we’ve learned most in coding that, that the rest of the world is not yet, I think ready for, I mean, we’re, they’ll, they’ll have to be ready for it because it’s just gonna inevitably happen is I think in coding.

What, what’s interesting is if you think about the practice of coding today versus two years ago. It’s probably the most changed workflow in maybe the history of time from the amount of time it’s changed, right? Yeah. Like, like has any, has any workflow in the entire economy changed that quickly in terms of the amount of change?

I just, you know, at least in any knowledge worker workflow, there’s like very rarely been an event where one piece of technology and work practice has so fundamentally, you know, changed, changed what you do. Like you don’t write code, you talk to an agent and it goes and [00:16:00] does it for you, and you may be at best review it.

And even that’s even probably like, like largely not even what you’re doing. What’s happening is we are changing our work to make the agents effective. In that model, the agent didn’t really adapt to how we work. We basically adapted to how the agent works. Mm-hmm. All of the economy has to go through that exact same evolution.

The rest of the economy is gonna have to update its workflows to make agents effective. And to give agents the context that they need and to actually figure out what kind of prompting works and to figure out how do you ensure that the agent has the right access to information to be able to execute on its work.

I, you know, this is not the panacea that people were hoping for, of the agent drops in, just automates your life. Like you have to basically re-engineer your workflow to get the most out of agents and, uh, and that, that’s just gonna take, you know, multiple years across the economy. Right now it’s a huge asset and an advantage for the teams that do it early and that are kinda wired into doing this.

‘cause [00:17:00] you’ll see compounding returns, but that’s just gonna take a while for most companies to actually go and get this deployed.

swyx: I love, I love pushing back. I think that. That is what a lot of technology consultants love to hear this sort of thing, right? Yeah, yeah, yeah. First to, to embrace the ai. Yes. To get to the promised land, you must pay me so much money to a hundred percent to adopt the prescribed way of, uh, conforming to the agents.

Yes. And I worry that you will be eclipsed by someone else who says, no, come as you are.

Aaron Levie: Yeah.

swyx: And we’ll meet you where you are.

Aaron Levie: And, and, and and what was the thing that went viral a week ago? OpenAI probably, uh, is hiring F Dees. Yeah. Uh, to go into the enterprise. Yeah. Yeah. And then philanthropic is embedded at Goldman Sachs.

Yeah. So if the labs are having to do this, if, if the labs have decided that they need to hire FDE and professional services, then I think that’s a pretty clear indication that this, there’s no easy mode of workflow transformation. Yeah. Yeah. So, so to your point, I think actually this is a market opportunity for, you know, new professional services and consulting [00:18:00] firms that are like Agent Build and they, and they kind of, you know, go into organizations and they figure out how to re-engineer your workflows to make them more agent ready and get your data into the right format and, you know, reconstruct your business process.

So you’re, you’re not doing most of the work. You’re telling agents how to do the work and then you’re reviewing it. But I haven’t seen the thing that can just drop in and, and kinda let you not go through those changes.

swyx: I don’t know how that kind of sales pitch goes over. Yeah. You know, you’re, you’re saying things like, well, in my sort of nice beautiful walled garden, here’s, there’s, uh, because here’s this, here’s this beautiful box account that has everything.

Yes. And I’m like, well, most, most real life is extremely messy. Sure. And like, poorly named and there duplicate this outdated s**t

Aaron Levie: a hundred percent. And so No, no, a hundred percent. And so this is actually No. So, so this is, I mean, we agree that, that getting to the beautiful garden is gonna be tough.

swyx: Yeah.

Aaron Levie: There’s also the other end of the spectrum where I, I just like, it’s a technical impossibility to solve. The agent is, is truly cannot get enough context to make the right decision in, in the, in the incredibly messy land. Like there’s [00:19:00] no a GI that will solve that. So, so we’re gonna have to kind of land in somewhere in between, which is like we all collectively get better at.

Documentation practices and, and having authoritative relatively up-to-date information and putting it in the right place like agents will, will certainly cause us to be much better organized around how we work with our information, simply because the severity of the agent pulling the wrong data will be too high and the productivity gain of that you’ll miss out on by not doing this will be too high as well, that you, that your competition will just do it and they’ll just have higher velocity.

So, uh, and, and we, we see this a lot firsthand. So we, we build a series of agents internally that they can kind of have access to your full box account and go off and you give it a task and it can go find whatever information you’re looking for and work with. And, you know, thank God for the model progress, but like, if, if you gave that task to an agent.

Nine months ago, you’re just gonna get lots of bogus answers because it’s gonna, it’s gonna say, Hey, here’s, here are fi [00:20:00] five, you know, documents that all kind of smell like the right thing. And I’m gonna, but I, but you’re, you’re putting me on the clock. ‘cause my assistant prompt says like, you know, be pretty smart, but also try and respond to the user and it’s gonna respond.

And it’s like, ah, it got the wrong document. And then you do that once or twice as a knowledge worker and you’re just never

swyx: again,

Aaron Levie: never again. You’re just like done with the system.

swyx: Yeah. It doesn’t work.

Aaron Levie: It doesn’t work. And so, you know, Opus four six and Gemini three one Pro and you know, whatever the latest five 3G BT will be, like, those things are getting better and better and it’s using better judgment.

And this sort of like the, all of these updates to the agentic tool and search systems are, are, we’re seeing, we’re seeing very real progress where the agent. Kind of can, can almost smell some things a little bit fishy when it’s getting, you know, we, we have this process where we, we have it go fan out, do a bunch of searches, pull up a bunch of data, and then it has to sort of do its own ranking of, you know, what are the right documents that, that it should be working with.

And again, like, you know, the intelligence level of a model six months ago, [00:21:00] it’d be just throwing a dart at like, I’m just, I’m gonna grab these seven files and I, I pray, I hope that that’s the right answer. And something like an opus first four five, and now four six is like, oh, it’s like, no, that one doesn’t seem right relative to this question because I’m seeing some signal that is making that, you know, that’s contradicting the document where it would normally be in the tree and who should have access.

Like it’s doing all of that kind of work for you. But like, it still doesn’t work if you just have a total wasteland of data. Like, it’s just not, it’s just not possible. Partly ‘cause a human wouldn’t even be able to do it. So basically if a, if a really, really smart human. Could not do that task in five or 10 minutes for a search retrieval type task.

Look, you know, your agent’s not gonna be able to do it any better. You see this all day long. So

Context Engineering and Search Limits

swyx: this touches on a thing that just passionate about it was just context engineering. I, I’m just gonna let you ramble or riff on, on context engineering. If, if, if there’s anything like he, he did really good work on context fraud, which has really taken over as like the term that people use and the reference

Aaron Levie: a hundred percent.

We, we all we think about is, is the context rob problem. [00:22:00]

Jeff Huber: Yeah, there’s certainly a lot of like ranking considerations. Gentech surgery think is incredibly promising. Um, yeah, I was trying to generate a question though. I think I have a question right now. Swyx.

Aaron Levie: Yeah, no, but like, like I think there was this moment, um, you know, like, I don’t know, two years ago before, before we knew like where the, the gotchas were gonna be in ai and I think someone was like, was like, well, infinite context windows will just solve all of these problems and ‘cause you’ll just, you’ll just give the context window like all the data and.

It’s just like, okay, I mean, maybe in 2035, like this is a viable solution. First of all, it, it would just, it would just simply cost too much. Like we just can’t give the model like the 5,000 documents that might be relevant and it’s gonna read them all. And I’ve seen enough to, to start believing in crazy stuff.

So like, I’m willing to just say, sure. Like in, in 10 years from now,

swyx: never say, never, never.

Aaron Levie: In, in 10 years from now, we’ll have infinite context windows at, at a thousandth of the price of today. Like, let’s just like believe that that’s possible, but Right. We’re in reality today. So today we have a context engineering [00:23:00] problem, which is, I got, I got, you know, 200,000 tokens that I can work with, or prob, I don’t even know what the latest graph is before, like massive degradation.

16. Okay. I have 60,000 tokens that I get to work with where I’m gonna get accurate information. That’s not a lot of tokens for a corpus of 10 million documents that a knowledge worker might have across all of the teams and all the projects and all the people they work with. I have, I have 10 million documents.

Which, you know, maybe is times five pages per document or something like that. I’m at 50 million pages of information and I have 60,000 tokens. Like, holy s**t. Yeah. This is like, how do I bridge the 50 million pages of information with, you know, the couple hundred that I get to work with in that, in that token window.

Yeah. This is like, this is like such an interesting problem and that’s why actually so much work is actually like, just like search systems and the databases and that layer has to just get so locked in, but models getting better and importantly [00:24:00] knowing when they’ve done a search, they found the wrong thing, they go back, they check their work, they, they find a way to balance sort of appeasing the user versus double checking.

We have this one, we have this one test case where we ask the agent to go find. 10 pieces of information.

swyx: Is this the complex work eval?

Aaron Levie: Uh, this is actually not in the eval. This is, this is sort of just like we have a bunch of different, we have a bunch of internal benchmark kind of scenarios. Every time we, we update our agent, we have one, which is, I ask it to find all of our office addresses, and I give it the list of 10 offices that we have.

And there’s not one document that has this, maybe there should be, that would be a great example of the kind of thing that like maybe over time companies start to, you know, have these sort of like, what are the canonical, you know, kind of key areas of knowledge that we need to have. We don’t seem to have this one document that says, here are all of our offices.

We have a bunch of documents that have like, here’s the New York office and whatever. So you task this agent and you, you get, you say, I need the addresses for these 10 offices. Okay. And by the way, if you do this on any, you know, [00:25:00] public chat model, the same outcome is gonna happen. But for a different kind of query, you give it, you say, I need these 10 addresses.

How many times should the agent go and do its search before it decides whether or not, there’s just no answer to this question. Often, and especially the, the, let’s say lower tier models, it’ll come back and it’ll give you six of the 10 addresses. And it’ll, and I’ll just say I couldn’t find the other

swyx: four.

It, it doesn’t know what It doesn’t know. It

Aaron Levie: doesn’t know what It doesn’t know. Yeah. So the model is just like, like when should it stop? When should it stop doing? Like should it, should it do that task for literally an hour and just keep cranking through? Maybe I actually made up an office location and it doesn’t know that I made it up and I didn’t even know that I made it up.

Like, should it just keep, re should it read every single file in your entire box account until it, until it should exhaust every single piece of information.

swyx: Expensive.

Aaron Levie: These are the new problems that we have. So, you know, something like, let’s say a new opus model is sort of like, okay, I’m gonna try these types of queries.

I didn’t get exactly what I wanted. I’m gonna try again. I’m gonna, at [00:26:00] some point I’m gonna stop searching. ‘cause I’ve determined that that no amount of searching is gonna solve this problem. I’m just not able to do it. And that judgment is like a really new thing that the model needs to be able to have.

It’s like, when should it give up on a task? ‘cause, ‘cause you just don’t, it’s a can’t find the thing. That’s the real world of knowledge, work problems. And this is the stuff that the coding agents don’t have to deal with. Because they, it just doesn’t like, like you’re not usually asking it about, you’re, you’re always creating net new information coming right outta the model for the most part.

Obviously it has to know about your code base and your specs and your documentation, but, but when you deploy an agent on all of your data that now you have all of these new problems that you’re dealing with

Jeff Huber: our, uh, follow follow-up research to context ride is actually on a genetic search. Ah. Um, and we’ve like right, sort of stress tested like frontier models and their ability to search.

Um, and they’re not actually that good at searching. Right. Uh, so you’re sort of highlighting this like explore, exploit.

swyx: You’re just say, Debbie, Donna say everything doesn’t work. Like,

Aaron Levie: well,

Jeff Huber: somebody has to be,

Aaron Levie: um, can I just throw out one more thing? Yeah. That is different from coding and, and the rest [00:27:00] of the knowledge work that I, I failed to mention.

So one other kind of key point is, is that, you know, at the end of the day. Whether you believe we’re in a slop apocalypse or, or whatever. At the end of the day, if you, if you build a working product at the end of, if you, if you’ve built a working solution that is ultimately what the customer is paying for, like whether I have a lot of slop, a little slop or whatever, I’m sure there’s lots of code bases we could go into in enterprise software companies where it’s like just crazy slop that humans did over a 20 year period, but the end customer just gets this little interface.

They can, they can type into it, it does its thing. Knowledge work, uh, doesn’t have that property. If I have an AI model, go generate a contract and I generate a contract 20 times and, you know, all 20 times it’s just 3% different and like that I, that, that kind of lop introduces all new kinds of risk for my organization that the code version of that LOP didn’t, didn’t introduce.

These are, and so like, so how do you constrain these models to just the part that you want [00:28:00] them to work on and just do the thing that you want them to do? And, and, you know, in engineering, we don’t, you can’t be disbarred as an engineer, but you could be disbarred as a lawyer. Like you can do the wrong medical thing In healthcare, you, there’s no, there’s no equivalent to that of engineering.

Like, do

swyx: you want there to be, because I’ve considered software

Jeff Huber: engineer. What’s that? Civil engineering there is, right? Not

Aaron Levie: software civil engineer. Sure. Oh yeah, for sure. But like in any of our companies, you like, you know, you’ll be forgiven if you took down the site and, and we, we will do a rollback and you’ll, you’ll be in a meeting, but you have not been disbarred as an engineer.

We don’t, we don’t change your, you know, your computer science, uh, blame

Jeff Huber: degree, this postmortem.

Aaron Levie: Yeah, exactly. Exactly. So, so, uh, now maybe we collectively as an industry need to figure out like, what are you liable for? Not legally, but like in a, in a management sense, uh, of these agents. All sorts of interesting problems that, that, that, uh, that have to come out.

But in knowledge work, that’s the real hostile environments that we’re operating in. Hmm.

swyx: I do think like, uh, a lot of the last year’s, 2025 story was the rise of coding agents and I think [00:29:00] 2026 story is definitely knowledge work agents. Yes. A hundred

Aaron Levie: percent.

swyx: Right. Like that would, and I think open claw core work are just the beginning.

Yes. Like it’s, the next one’s gonna just gonna be absolute craziness.

Aaron Levie: It it is. And, and, uh, and it’s gonna be, I mean, again, like this is gonna be this, this wave where we, we are gonna try and bring as many of the practices from coding because that, that will clearly be the forefront, which is tell an agent to go do something and has an access to a set of resources.

You need to be responsible for reviewing it at the end of the process. That to me is the, is the kind of template that I just think goes across knowledge, work and odd. Cowork is a great example. Open Closet’s a great example. You can kind of, sort of see what Codex could become over time. These are some, some really interesting kind of platforms that are emerging.

swyx: Okay. Um, I wanted to, we touched on evals a little bit. You had, you had the report that you’re gonna go bring up and then I was gonna go into like, uh, boxes, evals, but uh, go ahead. Talk about your genetic search thing.

Jeff Huber: Yeah. Mostly I think kinda a few of the insights. It’s like number one frontier model is not good at search.

Humans have this [00:30:00] natural explore, exploit trade off where we kinda understand like when to stop doing something. Also, humans are pretty good at like forgetting actually, and like pruning their own context, whereas agents are not, and actually an agent in their kind of context history, if they knew something was bad and they even, you could see in the trace the reason you trace, Hey, that probably wasn’t a good idea.

If it’s still in the trace, still in the context, they’ll still do it again. Uhhuh. Uh, and so like, I think pruning is also gonna be like, really, it’s already becoming a thing, right? But like, letting self prune the con windows

swyx: be a big deal. Yeah. So, so don’t leave the mistake. Don’t leave the mistake in there.

Cut out the mistake but tell it that you made a mistake in the past and so it doesn’t repeat it.

Jeff Huber: Yeah. But like cut it out so it doesn’t get like distracted by it again. ‘cause really, you know, what is so, so it will repeat its mistake just because it’s been, it’s in

swyx: the

Jeff Huber: context. It’s

Aaron Levie: in the context so much.

That’s a few shot example. Even if it, yeah.

Jeff Huber: It’s like oh this

Aaron Levie: is a great thing to go try even if

Jeff Huber: it didn’t work.

Aaron Levie: Yeah,

Jeff Huber: exactly.

Aaron Levie: So

Jeff Huber: there’s like a bunch of stuff there. Just

Aaron Levie: Groundhogs Day inside these models. Yeah. I’m gonna go keep doing the same wrong

Jeff Huber: thing. Covering sense. I feel like, you know, some creator analogy you’re trying like fit a manifold in latent space, which kind is doing break program synthesis, which is kinda one we think about we’re doing right.

Like, you know, certain [00:31:00] facts might be like sort of overly pitting it. There are certain, you know, sec sectors of latent space and so like plug clean space. Yeah. And, uh, and

swyx: so we have a bell, our editor as a bell every time you say that. So

Jeff Huber: you have, you have to like remove those, like

swyx: you shoulda a gong like TPN or something.

If

Jeff Huber: we gong, you either remove those links to like kinda give it the freedom, kind of do what you need to do. So, but yeah. We’ll, we’ll release more soon. That’s

Aaron Levie: awesome.

Jeff Huber: That’ll, that’ll be cool.

swyx: We’re a cerebral podcast that people listen to us and, and sort of think really deep. So yeah, we try to keep it subtle.

Okay. We try to keep it.

Aaron Levie: Okay, fine.

Inside Agent Evals

swyx: Um, you, you guys do, you guys do have EVs, you talked about your, your office thing, but, uh, you’ve been also promoting APEX agents and complex work. Uh, yeah, whatever you, wherever you wanna take this just Yeah. How you

Aaron Levie: Apex is, is obviously me, core’s, uh, uh, kind of, um, agent eval.

We, we supported that by sort of. Opening up some data for them around how we kind of see these, um, data workspaces in, in the, you know, kind of regular economy. So how do lawyers have a workspace? How do investment bankers have a workspace? What kind of data goes into those? And so we, [00:32:00] we partner with them on their, their apex eval.

Our own, um, eval is, it’s actually relatively straightforward. We have a, a set of, of documents in a, in a range of industries. We give the agent previously did this as a one shot test of just purely the model. And then we just realized we, we need to, based on where everything’s going, it’s just gotta be more agentic.

So now it’s a bit more of a test of both our harness and the model. And we have a rubric of a set of things that has to get right and we score it. Um, and you’re just seeing, you know, these incredible jumps in almost every single model in its own family of, you know, opus four, um, you know, sonnet four six versus sonnet four five.

swyx: Yeah. We have this up on screen.

Aaron Levie: Okay, cool. So some, you’re seeing it somewhere like. I, I forget the to, it was like 15 point jump, I think on the main, on the overall,

swyx: yes.

Aaron Levie: And it’s just like, you know, these incredible leaps that, that are starting to happen. Um,

swyx: and OP doesn’t know any, like any, it’s completely held out from op.

Aaron Levie: This is not in any, there’s no public data which has, you know, Ben benefits and this is just a private eval that we [00:33:00] do, and then we just happen to show it to, to the world. Hmm. So you can’t, you can’t train against it. And I think it’s just as representative of. It’s obviously reasoning capabilities, what it’s doing at, at, you know, kind of test time, compute capabilities, thinking levels, all like the context rot issues.

So many interesting, you know, kind of, uh, uh, capabilities that are, that are now improving

swyx: one sector that you have. That’s interesting.

Industries and Datasets

swyx: Uh, people are roughly familiar with healthcare and legal, but you have public sector in there.

Aaron Levie: Yeah.

swyx: Uh, what’s that? Like, what, what, what is that?

Aaron Levie: Yeah, and, and we actually test against, I dunno, maybe 10 industries.

We, we end up usually just cutting a few that we think have interesting gains. All extras, won a lot of like government type documents. Um,

swyx: what is that? What is it? Government type documents?

Aaron Levie: Government filings. Like a tax

swyx: return, like

Aaron Levie: a probably not tax returns. It would be more of what would go the government be using, uh, as data.

So, okay. Um, so think about research that, that type of, of, of data sets. And then we have financial services for things like data rooms and what would be in an investment prospectus. Uhhuh,

swyx: that one you can dog food.

Aaron Levie: Yeah, exactly. Exactly. Yes. Yes. [00:34:00] So, uh, so we, we run the models, um, in now, you know, more of an agent mode, but, but still with, with kinda limited capacity and just try and see like on a, like, for like basis, what are the improvements?

And, and again, we just continue to be blown away by. How, how good these models are getting.

swyx: Yeah, I mean, I think every serious AI company needs something like that where like, well, this is the work we do. Here’s our company eval. Yeah. And if you don’t have it, well, you’re not a serious AI company.

Aaron Levie: There’s two dimensions, right?

So there’s, there’s like, how are the models improving? And so which models should you either recommend a customer use, which one should you adopt? But then every single day, we’re making changes to our agents. And you need to know

swyx: if you regressed,

Aaron Levie: if you know. Yeah. You know, I’ve been fully convinced that the whole agent observability and eval space is gonna be a massive space.

Um, super excited for what Braintrust is doing, excited for, you know, Lang Smith, all the things. And I think what you’re going to, I mean, this is like every enter like literally every enterprise right now. It’s like the AI companies are the customers of these tools. Every enterprise will have this. Yeah, you’ll just [00:35:00] have to have an eval.

Of all of your work and like, we’ll, you’ll have an eval of your RFP generation, you’ll have an eval of your sales material creation. You’ll have an eval of your, uh, invoice processing. And, and as you, you know, buy or use new agentic systems, you are gonna need to know like, what’s the quality of your, of your pipeline.

swyx: Yeah.

Aaron Levie: Um, so huge, huge market with agent evals.

swyx: Yeah.

Building the Agent Team

swyx: And, and you know, I’m gonna shout out your, your team a bit, uh, your CTO, Ben, uh, did a great talk with us last year. Awesome. And he’s gonna come back again. Oh, cool. For World’s Fair.

Aaron Levie: Yep.

swyx: Just talk about your team, like brag a little bit. I think I, I think people take these eval numbers in pretty charts for granted, but No, there, I mean, there’s, there’s lots of really smart people at work during all this.

Aaron Levie: Biggest shout out, uh, is we have a, we have a couple folks at Dya, uh, Sidarth, uh, that, that kind of run this. They’re like a, you know, kind of tag tag team duo on our evals, Ben, our CTO, heavily involved Yasha, head of ai, uh, you know, a bunch of folks. And, um, evals is one part of the story. And then just like the full, you know, kind of AI.

An agent team [00:36:00] is, uh, is a, is a pretty, you know, is core to this whole effort. So there’s probably, I don’t know, like maybe a few dozen people that are like the epicenter. And then you just have like layers and layers of, of kind of concentric circles of okay, then there’s a search team that supports them and an infrastructure team that supports them.

And it’s starting to ripple through the entire company. But there’s that kind of core agent team, um, that’s a pretty, pretty close, uh, close knit group.

swyx: The search team is separate from the infra team.

Aaron Levie: I mean, we have like every, every layer of the stack we have to kind of do, except for just pure public cloud.

Um, but um, you know, we, we store, I don’t even know what our public numbers are in, you know, but like, you can just think about it as like a lot of data is, is stored in box. And so we have, and you have every layer of the, of the stack of, you know, how do you manage the data, the file system, the metadata system, the search system, just all of those components.

And then they all are having to understand that now you’ve got this new customer. Which is the agent, and they’ve been building for two types of customers in the past. They’ve been building for users and they’ve been building for like applications. [00:37:00] And now you’ve got this new agent user, and it comes in with a difference of it, of property sometimes, like, hey, maybe sometimes we should do embeddings, an embedding based, you know, kind of search versus, you know, your, your typical semantic search.

Like, it’s just like you have to build the, the capabilities to support all of this. And we’re testing stuff, throwing things away, something doesn’t work and, and not relevant. It’s like just, you know, total chaos. But all of those teams are supporting the agent team that is kind of coming up with its requirements of what, what do we need?

swyx: Yeah. No, uh, we just came from, uh, fireside chat where you did, and you, you talked about how you’re doing this. It’s, it’s kind of like an internal startup. Yeah. Within the broader company. The broader company’s like 3000 people. Yeah. But you know, there’s, there’s a, this is a core team of like, well, here’s the innovation center.

Aaron Levie: Yeah.

swyx: And like that every company kind of is run this way.

Aaron Levie: Yeah. I wanna be sensitive. I don’t call it the innovation center. Yeah. Only because I think everybody has to do innovation. Um, there, there’s a part of the, the, the company that is, is sort of do or die for the agent wave.

swyx: Yeah.

Aaron Levie: And it only happens to be more of my focus simply because it’s existential that [00:38:00] we get it right.

swyx: Yeah.

Aaron Levie: All of the supporting systems are necessary. All of the surrounding adjacent capabilities are necessary. Like the only reason we get to be a platform where you’d run an agent is because we have a security feature or a compliance feature, or a governance feature that, that some team is working on.

But that’s not gonna be the make or break of, of whether we get agents right. Like that already exists and we need to keep innovating there. I don’t know what the right, exact precise number is, but it’s not a thousand people and it’s not 10 people. There’s a number of people that are like the, the kind of like, you know, startup within the company that are the make or break on everything related to AI agents, you know, leveraging our platform and letting you work with your data.

And that’s where I spend a lot of my time, and Ben and Yosh and Diego and Teri, you know, these are just, you know, people that, that, you know, kind of across the team. Are working.

swyx: Yeah. Amazing.

Read Write Agent Workflows

Jeff Huber: How do you, how do you think about, I mean, you talked a lot about like kinda read workflows over your box data. Yep.

Right. You know, gen search questions, queries, et cetera. But like, what about like, write or like authoring workflows?

Aaron Levie: Yes. I’ve [00:39:00] already probably revealed too much actually now that I think about it. So, um, I’ve talked about whatever,

Jeff Huber: whatever you can.

Aaron Levie: Okay. It’s just us. It’s just us. Yeah. Okay. Of course, of course.

So I, I guess I would just, uh, I’ll make it a little bit conceptual, uh, because again, I’ve already, I’ve already said things that are not even ga but, but we’ve, we’ve kinda like danced around it publicly, so I, yeah, yeah. Okay. Just like, hopefully nobody watches this, um, episode. No.

swyx: It’s tidbits for the Heidi engaged to go figure out like what exactly, um, you know, is, is your sort of line of thinking.

Sure. They can connect the dots.

Aaron Levie: Yeah. So, so I would say that, that, uh, we, you know, as a, as a place where you have your enterprise content, there’s a use case where I want to, you know, have an agent read that data and answer questions for me. And then there’s a use case where I want the agent to create something.

And use the file system to create something or store off data that it’s working on, or be able to have, you know, various files that it’s writing to about the work it’s doing. So we do see it as a total read write. The harder problem has so far been the read only because, because again, you have that kind of like 10 [00:40:00] million to one ratio problem, whereas rights are a lot of, that’s just gonna come from the model and, and we just like, we’ll just put it in the file system and kinda use it.

So it’s a little bit of a technically easier problem, but the only part that’s like, not necessarily technically hard, it is just like it’s not yet perfected in the state of the ecosystem is, you know, building a beautiful PowerPoint presentation. It’s still a hard problem for these models. Like, like we still, you know, like, like these formats are just, we’re not built for.

They’re

swyx: working on it.

Aaron Levie: They’re, they’re working on it. Everybody’s working on it.

swyx: Every launch is like, well, we do PowerPoint now.

Aaron Levie: We’re getting, yeah, getting a lot, getting a lot of better each time. But then you’ll do this thing where you’ll ask the update one slide and all of a sudden, like the fonts will be just like a little bit different, you know, on two of the slides, or it moved, you know, some shape over to the left a little bit.

And again, these are the kind of things that, like in code, obviously you could really care about if you really care about, you know, how beautiful is the code, but at the end, user doesn’t notice all those problems and file creation, the end user instantly sees it. You’re [00:41:00] like, ah, like paragraph three, like, you literally just changed the font on me.

Like it’s a totally different font and like midway through the document. Mm-hmm. Those are the kind of things that you run into a lot of in the, in the content creation side. So, mm-hmm. We are gonna have native agents. That do all of those things, they’ll be powered by the leading kind of models and labs.

But the thing that I think is, is probably gonna be a much bigger idea over time is any agent on any system, again, using Box as a file system for its work, and in that kind of scenario, we don’t necessarily care what it’s putting in the file system. It could put its memory files, it could put its, you know, specification, you know, documents.

It could put, you know, whatever its markdown files are, or it could, you know, generate PDFs. It’s just like, it’s a workspace that is, is sort of sandboxed off for its work. People can collaborate into it, it can share with other people. And, and so we, we were thinking a lot about what’s the right, you know, kind of way to, to deliver that at scale.

Docs Graphs and Founder Mode

swyx: I wanted to come into sort of the sort of AI transformation or AI sort of, uh, operations things. [00:42:00] Um, one of the tweets that you, that you wanted to talk about, this is just me going through your tweets, by the way. Oh, okay. I mean, like, this is, you read

Aaron Levie: one by one,

swyx: you’re the, you’re the easiest guest to prep for because you, you already have like, this is the, this is what I’m interested in.

I’m like, okay, well, are

Aaron Levie: we gonna get to like, like February, January or something? Where are we in the, in the timelines? How far back are we going?

swyx: Can you, can you describe boxes? A set of skills? Right? Like that, that’s like, that’s like one of the extremes of like, well if you, you just turn everything into a markdown file.

Yeah. Then your agent can run your company. Uh, like you just have to write, find the right sequence of words to

Aaron Levie: Yes.

swyx: To do it.

Aaron Levie: Sorry, is

that

swyx: the question? So I think the question is like, what if we documented everything? Yes. The way that you exactly said like,

Aaron Levie: yes.

swyx: Um, let’s get all the Fortune five hundreds, uh, prepared for agents.

Yes. And like, you know, everything’s in golden and, and nicely filed away and everything. Yes. What’s missing? Like, what’s left, right? Like

Aaron Levie: Yeah.

swyx: You’ve, you’ve run your company for a decade. Like

Aaron Levie: Yeah. I think the challenge is that, that that information changes a week later. And because something happened in the market for that [00:43:00] customer, or us as a company that now has to go get updated, and so these systems are living and breathing and they have to experience reality and updates to reality, which right now is probably gonna be humans, you know, kinda giving those, giving them the updates.

And, you know, there is this piece about context graphs as as, uh, that kinda went very viral. Yeah. And I, I, I was like a, i, I, I thought it was super provocative. I agreed with many parts of it. I disagree with a few parts around. You know, it’s not gonna be as easy as as just if we just had the agent traces, then we can finally do that work because there’s just like, there’s so much more other stuff that that’s happening that, that we haven’t been able to capture and digitize.

And I think they actually represented that in the piece to be clear. But like there’s just a lot of work, you know, that that has to, you just can’t have only skills files, you know, for your company because it’s just gonna be like, there’s gonna be a lot of other stuff that happens. Yeah. Change over time.

Yeah. Most companies are practically apprenticeships.

swyx: Most companies are practically apprenticeships. Like

Jeff Huber: every new employee who joins the team, [00:44:00] like you span one to three months. Like ramping them up.

Aaron Levie: Yes. All

Jeff Huber: that tat knowledge

Aaron Levie: is

Jeff Huber: not written down.

Aaron Levie: Yes.

Jeff Huber: But like, it would have to be if you wanted to like give it to an Asian.

Right. And so like that seems to me like to be

Aaron Levie: one is I think you’re gonna see again a premium on companies that can document this. Mm-hmm. Much. There’ll be a huge premium on that because, because you know, can you shorten that three month ramp cycle to a two week ramp cycle? That’s an instant productivity gain.

Can you re dramatically reduce rework in the organization because you’ve documented where all the stuff is and where the answers are. Can you make your average employee as good as your 90th percentile employee because you’ve captured the knowledge that’s sort of in the heads of, of those top employees and make that available.

So like you can see some very clear productivity benefits. Mm-hmm. If you had a company culture of making sure you know your information was captured, digitized, put in a format that was agent ready and then made available to agents to work with, and then you just, again, have this reality of like add a 10,000 person [00:45:00] company.

Mapping that to the, you know, access structure of the company is just a hard problem. Is like, is like, yeah, well, you just, not every piece of information that’s digitized can be shared to everybody. And so now you have to organize that in a way that actually works. There was a pretty good piece, um, this, this, uh, this piece called your company as a file is a file system.

I, did you see that one?

swyx: Nope.

Aaron Levie: Uh, yes. You saw it. Yeah. And, and, uh, I actually be curious your thoughts on it. Um, like, like an interesting kind of like, we, we agree with it because, because that’s how we see the world and, uh,

swyx: okay. We, we have it up on screen. Oh,

Aaron Levie: okay. Yeah. But, but it’s all about basically like, you know, we’ve already, we, we, we already organized in this kind of like, you know, permission structure way.

Uh, and, and these are the kind of, you know, natural ways that, that agents can now work with data. So it’s kind of like this, this, you know, kind of interesting metaphor, but I do think companies will have to start to think about how they start to digitize more, more of that data. What was your take?

Jeff Huber: Yeah, I mean, like the company’s probably like an acid compliant file system.

Aaron Levie: Uh,

Jeff Huber: yeah. Which I’m guessing boxes, right? So, yeah. Yes.

swyx: Yeah. [00:46:00]

Jeff Huber: Which you have a great piece on, but,

swyx: uh, yeah. Well, uh, I, I, my, my, my direction is a little bit like, I wanna rewind a little bit to the graph word you said that there, that’s a magic trigger word for us. I always ask what’s your take on knowledge graphs?

Yeah. Uh, ‘cause every, especially at every data database person, I just wanna see what they think. There’s been knowledge graphs, hype cycles, and you’ve seen it all. So.

Aaron Levie: Hmm. I actually am not the expert in knowledge graphs, so, so that you might need to

swyx: research, you don’t need to be an expert. Yeah. I think it’s just like, well, how, how seriously do people take it?

Yeah. Like, is is, is there a lot of potential in the, in the HOVI?

Aaron Levie: Uh, well, can I, can I, uh, understand first if it’s, um, is this a loaded question in the sense of are you super pro, super con, super anti medium? I

swyx: see pro, I see pros and cons. Okay. Uh, but I, I think your opinion should be independent of mine.

Aaron Levie: Yeah. No, no, totally. Yeah. I just want to see what I’m stepping into.

swyx: No, I know. It’s a, and it’s a huge trigger word for a lot of people out Yeah. In our audience. And they’re, they’re trying to figure out why is that? Because why

Aaron Levie: is this such a

swyx: hot item for them? Because a lot of people get graph religion.

And they’re like, everything’s a graph. Of course you have to represent it as a graph. Well, [00:47:00] how do you solve your knowledge? Um, changing over time? Well, it’s a graph.

Aaron Levie: Yeah.

swyx: And, and I think there, there’s that line of work and then there’s, there’s a lot of people who are like, well, you don’t need it. And both are right.

Aaron Levie: Yeah. And what do the people who say you don’t need it, what are they

swyx: arguing for Mark down files. Oh, sure, sure. Simplicity.

Aaron Levie: Yeah.

swyx: Versus it’s, it’s structure versus less structure. Right. That’s, that’s all what it is. I do.

Aaron Levie: I think the tricky thing is, um, is, is again, when this gets met with real humans, they’re just going to their computer.

They’re just working with some people on Slack or teams. They’re just sharing some data through a collaborative file system and Google Docs or Box or whatever. I certainly like the vision of most, most knowledge graph, you know, kind of futuristic kind of ways of thinking about it. Uh, it’s just like, you know, it’s 2026.

We haven’t seen it yet. Kind of play out as as, I mean, I remember. Do you remember the, um, in like, actually I don’t, I don’t even know how old you guys are, but I’ll for, for to show my age. I remember 17 years ago, everybody thought enterprises would just run on [00:48:00] Wikis. Yeah. And, uh, confluence and, and not even, I mean, confluence actually took off for engineering for sure.

Like unquestionably. But like, this was like everything would be in the w. And I think based on our, uh, our, uh, general style of, of, of what we were building, like we were just like, I don’t know, people just like wanna workspace. They’re gonna collaborate with other people.

swyx: Exactly. Yeah. So you were, you were anti-knowledge graph.

Aaron Levie: Not anti, not anti. So

swyx: not non

Aaron Levie: I’m not, I’m not anti. ‘cause I think, I think your search system, I just think these are two systems that probably, but like, I’m, I’m not in any religious war. I don’t want to be in anybody’s YouTube comments on this. There’s not a fight for me.

swyx: We, we love YouTube comments. We’re, we’re, we’re get into comments.

Aaron Levie: Okay. Uh, but like, but I, I, it’s mostly just a virtue of what we built. Yeah. And we just continued down that path. Yeah.

swyx: Yeah.

Aaron Levie: And, um, and that, that was what we pursued. But I’m not, this is not a, you know, kind of, this is not a, uh, it’s

swyx: not existential for you. Great.

Aaron Levie: We’re happy to plug into somebody else’s graph.

We’re happy to feed data into it. We’re happy for [00:49:00] agents to, to talk to multiple systems. Not, not our fight.

swyx: Yeah.

Aaron Levie: But I need your answer. Yeah. Graphs or nerd Snipes is very effective nerd.

swyx: See this is, this is one, one opinion and then I’ve,

Jeff Huber: and I think that the actual graph structure is emergent in the mind of the agent.

Ah, in the same way it is in the mind of the human. And that’s a more powerful graph ‘cause it actually involved over time.

swyx: So don’t tell me how to graph. I’ll, I’ll figure it out myself. Exactly. Okay. All right. And

Jeff Huber: what’s yours?

swyx: I like the, the Wiki approach. Uh, my, I’m actually like, uh, you know, obviously I spent some my time at cognition, which, uh, you, you know very well.

Yep. And they’ve had a lot of success with Deep Wiki. Yeah. It powers a lot of Devrel and brain

Aaron Levie: super powerful.

swyx: And it’s super, it’s useful for humans, but it’s, oh my God, it’s useful for agents.

Aaron Levie: Yes. Tell me if you think I’m, I’m wrong on this, but, but not much of an access control structure issue?

swyx: No.

Aaron Levie: There’s like the whole, you get the whole code base and everybody gets to,

swyx: well, before, before I speak too much, there may be some enterprise controls on Sure.

The enterprise Deb offering that I’m not familiar with. Yeah. But yeah, I don’t, I don’t have any, anything on the public side. But, you know, I, I think like, almost like every agent should have its [00:50:00] own wiki that it’s updating and that’s. Persistent memory and yeah, that is a very weak knowledge graph.

Jeff Huber: Yeah.

swyx: And you, you could strengthen it if you want more structure, but you may not need it.

Jeff Huber: Markdown files, having links and wiki style. Right. Yep. Very effective. Right, Lindy?

Aaron Levie: Yep.

swyx: I like that. As a, as a just general pattern. Um, okay. So, uh, last couple questions. Sure. But feel free to jump on in or, or if you want any rants.

Um, I see you as a very interesting and, and unusual founder where, like, you’ve been in a business and you are, you’re both like, you’re off like of two worlds, like you’re of Silicon Valley, but you’re also of the Fortune five hundreds. And like, I feel like your kind of founder mode is very different from the Brian Chesky founder mode.

And I’m just kinda curious if you have like ref reflections on like how you operate as a founder,

Aaron Levie: what would his founder mode be?

swyx: Don’t delegate.

Aaron Levie: Ah, right. And what, how would you put me,

swyx: you do delegate. Ah,

Aaron Levie: okay. I, I, I, I see the, um, I think that I, I don’t know that Brian and I would be that far removed from each other when you get to the specifics.

swyx: Okay.

Aaron Levie: So there’s a whole bunch that I delegate, [00:51:00] 90%. Of the work that happens at Box is fully, you know, fully delegated. We’ve got great leaders running, running, all that stuff. It’s just too much for my brain to handle. And probably 70% of the work, I’m gonna make up all the numbers here, probably 70% of the work at Box or 70, 80% of the work at Box.

I only need to really look at about 5% of that for like, some high leverage decisions to be involved in, you know, what’s the marketing message that we think is gonna resonate with, with customers. So that’s a little bit of high leverage thing that, that, that we do in marketing. But most of marketing activities I don’t get involved in.

What’s our sales pitch? Maybe I’ll be involved in that a little bit. Or like what’s roughly the investments or push we’re gonna do in certain verticals. You know, that’s about 5% of like the total bandwidth of, you know, this, the, the key areas of sales or go to market. Okay. So like. 70, 80% of the company, I can just do about 5%.[00:52:00]

And then, and then just like operationally, we’ve got great leaders and they’re gonna execute on that, and we collaborate on the 5% anyway. It’s not like I’m just like making up a decision and, and saying to go and do it. Then there’s this part that is like the existential part of the business, which is if we don’t do this right, we’re out of business.

And, uh, by virtue of just being a founder, you get kind of sucked into that part of the work because you can feel it. Like, this is like, like you can just see how the AI tsunami could wipe you out if you make just 2, 3, 4, 5 wrong decisions in this space. Like couple wrong architecture decisions, couple wrong AI feature decisions, couple wrong API platform decisions, and, and you might be out of the game in a year from now and like, you just feel it in your bones.

You, you know, this, uh, like, it’s just like, like, like we feel this all day long in this space given what’s happening. Hmm. And so that, in that area. It’s, you can’t kind of delegate in a classic sense. You still need to make sure you’ve got great leaders and strong hires and people that, that are have high agency.

‘cause [00:53:00] they wanna be able to the own part of the, the strategy and the roadmap or else you can’t hire good people. But, but you know, there’s gonna be a lot of little micro forks in the road that they will compound to determine whether you’ve succeed or fail. And so your kind of founder energy just like automatically draws you into, into those because, because they are the determining decisions of, of your company’s future.

And that’s kind of where I spend my time and I, and you have to kind of, you know, do it in a collaborative way again, because if you are only dictatorial and just, you know, you just won’t, won’t eventually be able to hire the best people. ‘cause they won’t wanna work on that environment. But you also just can’t like.

Abdicate all the responsibility because the risks are, are just simply too high. Like, and so you have to somehow, obviously, add some value. And so the value I add is I’ve seen 20 years of this business, so I, I think I can kind of piece together what I expect the value propositions are gonna be and how customers will react to certain things.

So that’s what I can bring to the table. And then you have this kind of existential fear of, if I get it wrong, it’s all on me anyway. [00:54:00] I don’t get to blame, you know, you know, the engineer that was working on that project, like, it’s all, it’s, it’s, it’s my fault, right? Like at the end of the day, it’ll be my fault if it doesn’t work.

So by virtue of of that liability, uh, responsibility, you just get pulled into needing to make sure like it’s all going a according to, to kind of how you think it needs to end up. I don’t, I don’t know how Brian would answer that, I guess, but like I, I, yeah,

swyx: it’s a long essay. It’s an interesting essay.

People should go and compare and contrast your answer versus his, uh, I do think that, um, systems have a way of letting entropy get to them. Yep. And you, you, if you step away for too long, you need to have a way to like check in and go like, well, do I need to come back in? Or are we good? And people are gonna tell you things are good, but they’re not good.

Yes,

Aaron Levie: yes. A hundred percent.

swyx: Yeah.

Aaron Levie: And that’s actually, I’m, um, I’m a fan of actually process for the, that 70 to 80%.

swyx: Yeah.

Aaron Levie: So that 70 to 80% the process is you’re gonna do a, you know, a quarterly business review and you’re gonna have a brand check-in, and you’re gonna do [00:55:00] those, like, you’re gonna make sure that, that you’re seeing all the, the right episodes of, of what’s changing and, and how, and how it’s kind of, you know, evolving and, and make sure it’s kind of going the right direction.

And then there’s some areas which is like, no, it’s 24 7. Like, like I guarantee after this podcast at 11:00 PM I’ll be doing a Zoom with Ben, uh, and probably some other people. ‘cause we’re gonna be talking about agents and, and new platform features and like, that’s amazing. That’s your just in the cauldron, you know, kind of grinding on, on, on that side.

swyx: Yeah. Yeah. That’s, uh, that’s extremely, um, realistic. Yeah. What is, what it’s like, and I just want to have people hear your perspective on what,

Token FOMO Culture

Aaron Levie: and this is what you like, and this is the, this is this like, um, you read the post about, you know, everybody having agents running on the weekend and, um, and it’s like, uh, you know, you, you just.

I mean, first of all, anybody crazy enough to come to Silicon Valley? Like we don’t bring good news about the sort of like healthiness of our environment right now. Like, like, like you have to,

swyx: and

Aaron Levie: [00:56:00] you have to know what you’re signing up for. But like, like, you know, there, there’s a real issue, which is like, shoot, do I have enough agents running?

And, and

swyx: oh yeah, I made a meme that was like semi viral for me about this. Exactly, yes. That was incredible. That’s,

Aaron Levie: and, and, and that, that

swyx: was, you can’t even enjoy a party these days. Becausecause, you’re working with your tokens.

Aaron Levie: No. You just compute out there that you’re not utilizing,

swyx: what the hell? Like,

so

Aaron Levie: like there’s

swyx: ad I paid for the $200, I’m gonna spend the $200.

Aaron Levie: Yeah.

swyx: Uh, I’m gonna spend $6,000 out of 200 bucks. Yeah, exactly.

Jeff Huber: Exactly.

Aaron Levie: We

Jeff Huber: need to make anthropic very unprofitable. So,

swyx: yeah. Yeah. We’re not doing a good enough job. Cool.

Production Function Secrets

swyx: I have a closing question. If you, unless you,

Jeff Huber: I have a question. I’ve asked this question in private before, but I ask it again, which is, uh, it’s a question that Tyler Cowen asks his guests on his podcast, which is, uh, what is the Aaron Levy production function?

And, uh, uh

swyx: Oh, I love

Jeff Huber: that. I love this question because there’s so a few people that I think are good at both executing. Also like distilling and like, just putting good ideas into the ether. Mm-hmm. You put a lot of good ideas into the ether. And so like what is the air levee production function that allows you you to do that versus others?[00:57:00]

Aaron Levie: How do I get that information? Or

swyx: I, I can give you a, a, a variant. Yeah. Which is what goes into air and levee.

Aaron Levie: Yeah.

swyx: And what goes out and how does it turn inside? Yeah.

Aaron Levie: I’m just trying to think of, ‘cause I mean, you know, there’s some very, I, I just read a lot of Twitter, uh, as well. And so like, I just, and you’ve, you

swyx: spent a lot of effort

Aaron Levie: too.

Jeff Huber: Contrast, you don’t see like, great. Many essays from Brian Chesky every day.

Aaron Levie: Uh,

Jeff Huber: but you

Aaron Levie: do

Jeff Huber: from you.

Aaron Levie: Oh, yeah. And you’re

Jeff Huber: kind of weird in that way, so

Aaron Levie: why? Maybe he’s, he, maybe he’s healthier than me. Actually. We should just like, we should just text him to see if, you know, he’s got a more I think he does

swyx: work out.

Aaron Levie: Yeah. He got bigger

swyx: muscles.

Aaron Levie: That’s the thing. I, I work out less than him and I tweet more than him. So, so that’s the, that’s how we’re balancing things out. I am, um, I mostly, the way I just think about it is, uh, is just, um, you know, there’s, there’s lots of work that’s happening in the business. I am getting to see the, all the problems that we are running into constantly.

And I am trying to, uh, be a little bit of a, create a flywheel between what we’re doing [00:58:00] internally, what, what, what. Then we talk about, uh, getting a feedback loop on that and seeing other people’s, you know, experiences of what they’re doing. Bring that back into the business. And, and so I just see, uh, like my job is as, you know, hopefully being able to kind of connect the dots.

Of, of what’s going on in the world with what’s going on in box. And then I just happened to tweet about that along the way.

swyx: Yeah.

Aaron Levie: Um, because

swyx: it’s all you, there’s no like,

Aaron Levie: yeah.

swyx: Editor,

Aaron Levie: there’s no,

swyx: yeah.

Aaron Levie: Yeah.

swyx: Wow.

Aaron Levie: The, uh, I got, um, there was a funny, uh, uh, my, I, I tried to get an internship in, um, between freshman and sophomore year of this company, and it was a, it was a film, uh, kind of production company in New York.

And, uh, I got the internship and then I emailed my liaison kind of guy who sponsored me for the internship and I said, Hey, I’d like to do a blog of my summer internship. Hmm. Where I blog about, you know, the, the being an intern at a production company in New York and. About like a, I dunno, half a day, a day later, [00:59:00] uh, they emailed me back saying they’ve rescinded the internship.

swyx: No.

Aaron Levie: Um, uh, yeah, because, because I showed a lack of judgment on, you know, professionalism, you know, or whatever. Like, like just even the, the idea that I would ask that question, red flags went up of like, who the f**k is this guy? So anyway, I, I only say that to say that like, like to me, just like, you know, building in public is just like a natural, is a natural thing.

And so I, so I just, you know, go through the day. We, we deal with interesting problems. I tweet about them. I get information back in the process. I, I see your work. I see your work. You know, I see a bunch of folks and, and try and, you know, kind of incorporate that back in the box. My job is to try and connect all these things together and, uh, and make, make it useful.

swyx: And you’re, I mean, you’re the number one spokesperson, right? So you do have to be out there.

Aaron Levie: Yeah, I, but I, I kind of would be doing it whether or not, like it’s, I don’t really think of it as a job requirement as much as like, I just like, I like social media.

Jeff Huber: You’re so good at it.

Aaron Levie: Yeah.

Jeff Huber: It’s so hard to believe.

So like,

Aaron Levie: okay, sorry.

Jeff Huber: Do you get up at 5:00 AM [01:00:00] with coffee? Is that your secret? It’s like, how do you work or do you actually just like, in the back of Waymo’s, like, is, do you do it that way? Like how do you do this?

Aaron Levie: It’s, it’s, no, it’s, it’s, it’s mostly that though. It’s mostly, uh, there’s a, you know, I, I, I have a commute home each night.

I try and see, you know, my kids’ most, most weekdays before I have to hop back online. So there’s like a 20 minute window there.

Jeff Huber: Okay.

Aaron Levie: Where I can kinda like distill the information that’s happened and nice. And be like, ah, is there anything I learned today that would be interesting to throw out there? Or anything that I saw.

And then probably somewhere between like seven 30 and 9:00 PM I finally get a chance to like look through the feed. Mm. And see like, did anything crazy happen in ai? And, um, uh, and then that’s, that will also kind of catalyze, you know, something Yep. As like, that’s the best I can kind of,

swyx: you

Aaron Levie: know, respect.

Yeah. Okay. Thanks.

swyx: Uh, and now I know you, you cut off his 8:00 PM I will try to get AI news out before 8:00 PM so I can help him.

Aaron Levie: Yeah.

swyx: Do, do his thing.

Aaron Levie: Ba basically, if, if I [01:01:00] don’t see it before eight to eight 30, I’m not gonna

swyx: Yeah. It’s, I’m gonna

Aaron Levie: be able to like court tweet or something.

swyx: Yeah,

yeah.

Aaron Levie: Uh, because, uh, because then I’m back on Zoom after that,

Film Roots to Box

swyx: so I wasn’t gonna plan on asking this, but you’ve mentioned, uh, you mentioned the film stuff.

Aaron Levie: Yeah.

swyx: And I know from one of my favorite parts of doing your research on you was that, uh, you got the idea for Box from like, the, the Paramount lot. Yeah. Uh, pushing paper. Uh, are you film guy? You, you’re a big,

Aaron Levie: uh, I, I I, I, I would say I used to be more of a film guy.

swyx: Yeah. What, what’s your, what what, what are your favorites?

If you have, you wanna list off any

Aaron Levie: kind of the classic, uh, wannabe film student classics are, are you

swyx: talking Scorsese?

Aaron Levie: Yeah. Panino, pop Fiction, Magnolia. Requiem for a dream, basically. Like if there was an art house film in the nineties, uh, to early two thousands, that was my genre. Yeah. That got me into like, wow, wouldn’t it be cool to do, you know, you know, film.

And then I, I thought maybe I could connect digital into it. Like, could you, could you do film online? That just seemed too [01:02:00] hard from a licensing standpoint. And then obviously Netflix, you know, kind of existed. Um, so I, I never quite was able to fully connect the dots on these things. But the internship at Paramount, um, was one kind of catalyst for starting box because we were using just traditional enterprise software.

And I was like, wow. It’s like really hard to share data, you know, just like files going back and forth. Um, but the same thing was happening in school as well, and so that all led to, led the box basically.

swyx: Um, well, a 24 is, uh, you know, kind of giving back the sort of resurgence of the independent film, I guess a

Aaron Levie: hundred percent.

swyx: Um, uh, in, in, in, in the face of all the Marvel slop.

Aaron Levie: Uh, you know, I was thinking about this the other day, and a 24 is, you know, uh, certainly the best, uh, EE example I’m sure of, of this today. But, um, you know, they just don’t, you know, you, it’s hard to make a film, uh, like, you know, no country for old men or, um, there will be blood like, like what is that movie today?

swyx: Yeah.

Aaron Levie: Like what is a brand new movie that is just like original? [01:03:00] You just watch it and you’re like, what, what did I just watch? So

swyx: my, my, you know, sixes movie bench is, uh, Forrest Gump.

Aaron Levie: Okay.

swyx: Which iconic in its time.

Aaron Levie: Yep. A hundred percent.

swyx: Never again.

Aaron Levie: Yeah. Yeah. We, we did not make, we don’t know how to make Fors Gump anymore.

Um, they will try it with the sequel

Jeff Huber: though, at some point.

Aaron Levie: For sure. I, I honestly fors

swyx: Gump two in 30

Aaron Levie: years. I’ll be fine with it. No, that Fors Gump has a kid. Like he’s still right. Yeah, he’s still right. Exactly. Um, I think for Gump has a grandkid would be like a good movie. Like what is the grandkid of Forres Gump doing in, uh, in 2026

swyx: goes tropical.

Aaron Levie: Yeah. But, um, yeah, I definitely, let’s, I wanna see good, I wanna see more movies out there.

AI Future of Movies

Aaron Levie: You know, I’m a little bit conflicted on AI and film because,

swyx: oh, that, let’s see that.

Aaron Levie: Well, because I, uh, the world does not need more slop on, on AI entertainment, but I’m kind of like in a mode where I think that AI is, is, is gonna be, you know, generally a pure positive.

Because if I’m a, [01:04:00] if I was me 25 years ago in high school, for sure, I would be making a full production film. That had explosions and car chases and, but then there’d be like people that would show up there. So like I think that ability to, to just, you get to be Spielberg, you know, is, is, you know, completely amazing and, and democratizing.

That is incredible. And I, you know, I’m, I’m concerned about like, how do you make sure that we still get PT Anderson. Along the way and, and can we make sure that those, those guys exist? And then interestingly, I never, and I never saw it, but Darren Aronofsky, I, I believe, has either put out or gonna put out a, an AI film, you know, even some of the best artists are, are, you know, starting to adopt this.

But, um, uh, but yeah, I, I definitely don’t want to, what I don’t wanna do is just be like in this like TikTok feed of just films and it’s just like, oh, this film about the car chase that does this thing. And it says like, we don’t need that. Like, like, [01:05:00] like this should be a form of entertainment and art and let’s use AI to accelerate the production process.

Do the really hard CG work that, that you just, you had to spend way too much money on previously to do the, you know, kind of like, let’s, let’s use it to test out all new kind of plot ideas. Uh, yeah. Previs.

Jeff Huber: Yeah, exactly. Like

Aaron Levie: backgrounds and that’s incredible. Like whatever. Yeah. And all those things are super incredible.

I still like the, it’s very nostalgic, but I still like the idea of like. This is a camera and a person and a person that says, you know, action. Uh, and then, and let’s hopefully like surround AI around that. Yeah. We’ll, but we’ll, we’ll see how that plays out.

swyx: Yeah. I think, you know, so one of the things that stability ai, uh, made an impression on me was like, well, you know, and at least now we can remake Game of Throne Season eight, and I can, you know, uh, like, like it was meant to be not, uh, not rushed.

Yeah.

Aaron Levie: And then you watch, um, well I have a six and a half year old and I, you know, you see a lot of these kid movies and you’re like, yeah, that probably will be ai. I don’t totally know the job math ‘cause I don’t know how many animators there are today. [01:06:00] But I actually think, weirdly, I think we could be producing more high quality, maybe even slightly educational kids entertainment.

And so it’s maybe that’s a positive is like we could just have like more, like you could just have a Pixar for like, you know, things where kids learn stuff. And it used to be these like very, you know, lo-fi uh, you know, kinda lesson things.

swyx: I mean, we had tellies, you know, that so slow.

Aaron Levie: So, so we, we could have way more of that.

And, and maybe every animator that today is making a Pixar film is now, you know, we’re like, we fragment that out and uh, but now they’re responsible for more content and they’ve got AI agents running. So like, so, so I think there’s some optimistic scenarios on the entertainment side is like, there’s a lot of great use cases for being able to do, you know, generative media.

swyx: Yeah. Yeah. Edu edutainment as well.

Media DevRel and Engineering

swyx: I guess one question I is, it’s kind of like a self-serving one and almost like an advice, uh, side of the, the, the, the question, one of the things I just, uh, really enjoyed, uh, researching you was that, uh, Michael Arrington had some influence in the [01:07:00] box journey because he went to his house party.

Aaron Levie: Yes.

swyx: And, and that’s how you got funding.

Aaron Levie: Yes.

swyx: One of latent spaces. That’s a deep cut, right?

Aaron Levie: Yeah. Very deep cut. That’s a oh six deep cut.

swyx: Yeah. Uh, do, I mean, do you want to tell that story? I don’t know if you’ve told it very

Aaron Levie: much. It’s not very much of the story. Yeah. Uh, because I probably just,

swyx: it’s like a random intro, right?

Like,

Aaron Levie: um, well, it was just he used to have house parties. Yeah. Uh, TechCrunch had had these house parties and, and it was, um, probably no different than somebody’s doing a house party in sf Uh, you know, just go, yeah. And you just go and you meet the VCs and founders and like, I’m gonna make up examples, so I don’t want to like, you know, there’d be like Chad Hurley over there pitching his, you know, YouTube to people.

And like, like that’s just like how it worked. And it was just like, wow. Like that was this era where all these new companies were, were emerging. And I met, uh, our first investor, uh, in Silicon Valley at one of these house parties, Emily Melton, who then brought us into D-D-D-F-J-D, that, that became our Series A.

So that was all because of Arrington’s, uh, backyard Party.

swyx: One of my inspirations for late space is to be as helpful, influential, whatever as TechCrunch was. That’s [01:08:00] awesome. In the day.

Aaron Levie: That’s Yeah.

swyx: What would a new TechCrunch today look like? You know, what, what, what, what should I, what should I do? I think there used to be TechCrunch Disrupt.

Yeah. You know, I could do that with my conference, but I haven’t done it yet.

Aaron Levie: Well, I mean, I think,

swyx: um, useful. I don’t know.

Aaron Levie: Uh, well, you know, actually interestingly, I would, I would argue Disrupt came after the period that was the, was that Deep cut period. Okay. So, so I think Di Disrupt, you know, ended up being, you know, you know, catalyzing.

I don’t even, I think Cloud Flare launched It disrupted, yes. Is that the story? Right.

swyx: They were runners up.

Aaron Levie: Okay. Okay. So like, so like, I think anytime. Anytime you can be in a, a launchpad is, is just great because it draws in people that are, that’s what I’m trying to do in that creative moment. And whether it needs to be a contest or, or just like everybody gets like five minutes and you’re fundraising.

I mean, who knows? But, but I mean, for what it’s worth, like, I don’t know, have that much advice. ‘cause I think you, you’re, you’re already doing it effectively. Like I, I just like watched the YouTube videos late at night. Um, uh, from the events. I haven’t [01:09:00] been to one of your events, but like from the, from the camera angles, it looks like everybody’s there trying,

Jeff Huber: trying.

So

Aaron Levie: what’s great is that people are gonna be in the audience as like two random people and they’ll be like, you know, the next, the next big AI company will come from, you know, people coming to a meetup. ‘cause they were like, ah, I came in from Chicago and I’m ah, from, you know. Poland and let’s go do a startup.

Like that’s

swyx: the

Aaron Levie: magic

swyx: of

Aaron Levie: the valley.

swyx: Dix Hy found his co-founder at a IE Oh, and I know of at least one marriage. That’s, that’s, wow,

Aaron Levie: you have marriages

swyx: already. Yeah. Yeah.

Aaron Levie: I

Jeff Huber: don’t,

Aaron Levie: I never heard that about,

swyx: that’s my go, that’s my favorite. KPI.

Aaron Levie: Wow. We have AI marriages at the, at the AI engineer conferences.

These are both

Jeff Huber: humans. To be clear,

swyx: that’s a very good clarification. I like that. You have to check.

Jeff Huber: Yes. That’s a

swyx: very good

Aaron Levie: clarification.

swyx: No, but I, I think you have, you’re, you’re insightful business leader with like, a lot of thoughts on media, so I just figured I would,

Aaron Levie: I mean, media is such an interesting space right now because, because I, you know, with the go direct model, every company is gonna have to be a media company.

You

swyx: are going, you are the og. Go direct.

Aaron Levie: Yeah. But, but, but you know, we [01:10:00] we’re, we’re still like. Like, I think, I think what, what you guys are doing, and I don’t even know all the overlapping relationships, but like I watch your guys’ videos of your events, watch your event videos, but like, it’s clearly like this is the new format, right?

Companies have to become channels to communicate with audiences. Yeah. I think the resurgence, resurgence maybe is a bad word ‘cause it implies it decline, but like, Devrel is hot. Yeah. Like the hottest thing of all time right now. I like if you could produce a fricking factory of Devrel people, like there’s just like unlimited jobs right now on the other end of that.

Yeah.

Jeff Huber: Yeah.

Aaron Levie: Um, ‘cause we’re gonna, everybody needs their services and APIs to be used by agents. And so we have to all find a way to like, like, Hey, look at me. Like, like agent over, oh please come over here agent. And that’s gonna, that’s a content game. Like how do you get the agents to see your stuff

swyx: Yeah.

Aaron Levie: And know your APIs and like, this is like a new world that, that we are in. And uh, it’s gonna be a. It’s, it’s gonna completely be a [01:11:00] digital marketing, you know, kind of world that we’re in.

swyx: Yeah. Uh, for what it’s worth, I’m trying to help by doing little writing bootcamps and basically turn into a Devrel bootcamp.

Um, where, you know, well, it’s a demand and supply problem. There’s, there’s huge demand. Yeah. There’s no supply. Wow. All this increase

Aaron Levie: supply. Why is your no supply?

swyx: The one, the really good ones were for themselves.

Aaron Levie: Uh huh.

swyx: Creator economy screwed, screwed you over.

Aaron Levie: So, so I see so, so Substack and Yes. YouTube payouts.

And that’s, is that

swyx: really making Patreon? Yeah. Like the, the most talented guys are making, you know, millions and just working for themselves while for you,

Aaron Levie: that’s not, we don’t want them to make that much money. Okay.

swyx: We need to be able to hire

Aaron Levie: people.

swyx: I mean, I think, I think like, you know, do do what some companies are doing, you know, I’m not saying it’s my situation exactly, but like give them equity and like Uhhuh it should probably would be worth more, uh, just like sort of helping them out.

Aaron Levie: Well, they are getting Oh, sorry. As full-time employees or not?

swyx: I’m part-time.

Aaron Levie: You need full-time.

swyx: I’m part-time.

Aaron Levie: Yeah. But, but you’re, you’re you n of one, like, we like also people that are full-time.

swyx: Yeah. Yeah. My classic joke or, or like, observation [01:12:00] was like, this was when HubSpot bought, like their, they bought like a newsletter business.

Uh, and then they bought the, my first million, like the, the sort of podcast. Oh, okay. Dharmesh, you must know Dharmesh. Um, so he’s like obsessed with this guy. Okay. So, so my conclusion was like every company must either build or buy a media company. Yes. Right. And until you, unless you realize that. You have to take it that seriously that you are running a media business in your company.

Yes. You will never be good at it.

Aaron Levie: Yes, a hundred percent.

swyx: Yeah.

Aaron Levie: Yeah. No, we’re, we’re very much taking that seriously. But, but still, and yet Devrel, I mean, I gotta do one plug. I don’t all is out. Please, please. We’re hiring a Devrel.

swyx: Yeah.

Like,

Jeff Huber: like please

swyx: no, all engineers here. Like, yeah. Like you’ve made it, like, and I just said every, every agent needs a box.

Like, let’s go, let’s go.

Aaron Levie: Thank you. No, that, that’s the headline. And we are hiring Devrel to make that happen. Uh, but yeah, I think Devrel is like the future job. So we’re all just gonna be doing Devrel in some form.

swyx: Okay. Yeah.

Aaron Levie: I mean, what is FD

swyx: developers are ruling the earth. Yeah.

Jeff Huber: What is FDI don’t know. Um,

Aaron Levie: no, it’s, it’s Devrel.

swyx: Yeah. Okay.

Aaron Levie: No, you just, you’re going to

swyx: a company, isn’t it just like glorify consulting? That’s, that’s the downside.

Aaron Levie: Sure. I mean, I guess nobody can like actually [01:13:00] d you know, fully define this, but, um, uh, but I think it’s, it’s, it’s micro Devrel, like you’re in the company, you’re helping them with the services.

Yeah. You’re doing a little bit extra implementation. Yeah.

swyx: Yeah.

Aaron Levie: Um, but, uh, but yeah, so it’s, uh, I, I think we’re all, you know, the thing that’s gonna happen on the ledger of software is we’re gonna produce far more output of code and thus features per dollar. But on the other end of this, we’re gonna actually end up spending probably just as much on how do you get all of that stuff to the customer, and it’s gonna create a new set of roles that we are all doing, partly because I, either, because there’s so much choice and now you have to kind of fight for attention there, or because this stuff is, is just changing so quickly that you have to technically help your customers.

Along the journey. Yeah, so, so I just think like, I, this is why I, I, I always laugh when, you know, people say you don’t need to be an engineer, don’t do computer science. I actually think like that is like still one of the most protected job categories because [01:14:00] things are only getting more technical. Things are only gonna get harder and anybody in a technical position is in the best position.

Yeah. To get agents deployed, get them built, get them adopted, build the, the, the custom code software to the, for the IT system, all of that.

swyx: So, yeah. Yeah. My, my classic founding story of like why I picked AI engineer as a title and as, as a, as a theme for this podcast as theme for my conference was, um, back in like early 2023, someone al came to me and said like, I’m all in on ai.

What should I do? And I was like, I just looked at her. I was like,

Jeff Huber: God dammit, there’s nothing you can do.

swyx: Like engineers are about to get so much more powerful than you Uhhuh. You don’t even understand.

Aaron Levie: Tell me there’s a good, did she go and then learn?

swyx: No, I didn’t, I didn’t say any of that to her.

Aaron Levie: Oh, oh, I see, I see, I see.

swyx: Okay. Yeah, I’m not, I’m not that honest. Well,

Aaron Levie: I hope, I hope somewhere out there. She, she did, went to some online academy.

swyx: Exactly. Learn to code.

Aaron Levie: Yeah.

swyx: But there, there’s a lot of people, like, there’s a lot of people who believe AI too much, and then they’re like, well, you don’t need to learn to code, so I won’t learn to code.

Yeah. And then there’s, there’s like, there’s a bunch of us who are like, just in that [01:15:00] sweet spot of like, we can code and we can wield AI a thousand times more effectively than you can. Yeah. And like, well, who’s gonna win here? Like

Jeff Huber: I, I think I, this was another, uh, a tweet, but it was like the observation that like, really software engineering for the past 30 years was the primary career track for like technical, high agency people that wanted to have a large outsize impact on the world.

swyx: Yeah.

Jeff Huber: And like, software was a means to, you know, do that Right. Effectively. Um, and so yeah, with ai, is it like that, uh, and, and for AI could eat software engineering or software engineering could eat all their kind of domains of discipline.

Aaron Levie: You, those pr same principles then get applied to every other and then function,

Jeff Huber: right?

Aaron Levie: Yeah, exactly. Yeah. I

Jeff Huber: mean, g team engineering, is that a hundred percent Anything else? Yeah.

Aaron Levie: Well, this is the, you know, uh, anybody who believes that an enterprise, and I’m, I’m, I’m mixed on the, I’m mixed on this is, but if you believe that an enterprise is going to build its own software for all of its problems, then you must be the most long on computer science, you know, as a discipline of all time, because guess what, most of the economy does not have enough engineers to then [01:16:00] maintain all those systems, to update to all those systems, to figure out the, the relationship between the business problem and what the code needs to do to go and actually manage that.

And so, so like that’s, that’s a very pro. Engineering job argument of what the future’s gonna look like. I’m still, again, I go back and forth on like, are you gonna really build all these things versus no prepackaged software, but no matter what, there’s gonna be 10 to a hundred times more code. So I think you can be very long engineering right now as just a, you know, purely on the dimension of, of software’s gonna become increasingly more important once agents are, are, you know, turning everything into software.

swyx: Yeah. All right. Three software guys say software in room. Okay.

Aaron Levie: Not biased at all. Okay.

swyx: But, uh, Aaron, your inspiration. All right. Take you. It’s such a pleasure.

Aaron Levie: All right. Good to be here.

AIE Europe CFP and AIE World’s Fair paper submissions for CAIS peer review are due TODAY - do not delay! Last call ever.

We’re excited to welcome METR for their first LS Pod, hopefully the first of many:

METR are keepers of currently the single most infamous chart in AI:

But every Latent Space reader should be sophisticated enough to know that the details matter and that hype and hyperbole go hand in hand in AI social media, because the millions of impressions that got, by people who don’t understand or care about the nuances, disclaimers, and error bars, far outreaches the 69k views on the corrections by the people who actually made the chart:

There’s a lot of nuance both in making benchmarks (as we discovered with OpenAI on our SWE-Bench Verified podcast) and in extrapolating results from them, especially where exponentials and sigmoids are concerned. METR’s Long Horizons work itself has known biases that the authors have responsibly disclosed, but go far too underappreciated in the pursuit of doomer chart porn.

If you’re interested in a short, sharable TED talk version of this pod, over at AIE CODE we were blessed to feature Joel twice, as a stage talk and with a longer form small workshop with Q&A:

We also make sure cover some of METR’s lesser known work on Threat Evaluation but also Developer Productivity, where 2x friend of the pod and now Zyphra founder Quentin Anthony was the ONLY productive participant!

Finally, if you’re the sort to read these show notes to the end, then you definitely deserve some pictures of Joel shredding the guitar at Love Band Karaoke which we mention at the end:

Full Video Pod

Timestamps

00:00 What METR Means00:39 Podcast Intro With Joel01:39 ME vs TR03:33 Time Horizon Origin Story04:56 Picking Tasks And Biases09:13 Time Horizon Misconceptions11:37 Opus 4.5 And Trendlines14:27 Productivity Studies And Explosions29:50 Compute Slows Progress30:47 Algorithms Need Compute32:45 Industry Spend and Data34:57 Clusters and Shipping Timelines36:44 Prediction Markets for Models38:10 Manifold Alpha Story43:04 Beyond Benchmarks Evals51:39 METR Roadmap and Farewell

Transcript