Inside Anthropic Labs: Mike Krieger on Claude Fable 5

Source material

Full source text

(upbeat music) - Mike, welcome to the show.

- Great to be here, Dan, good to see you.

- So for people who don't know you, you're the head of Enthropic Labs and you're the co-founder of Instagram.

And today what I want to talk to you about is Fable 5.

So Fable 5 is dropping tomorrow, we're recording this the day before, this will come out after it drops.

But what I really wanted to do is bring you on the show to tell me about what it's like to use this model beyond the first day.

I think we want to model this powerful drops.

It's so useful to have someone who's using it day in and day out to tell you this is where it's powerful, this is what it actually changes, this is what it doesn't change.

So that you kind of don't get the same AI psychosis type thing, you can actually think about, okay, this is how it fits into my life.

- Yeah, absolutely.

And it's also just been interesting.

We've had some models in this Mythos class leading up to the Fable release for a couple of months now.

And I think it's very exciting to see how people will build with this externally.

But I think you're also right that day one impressions, I think it really comes from getting to use this over a couple of weeks.

I think we've seen that even with previous models, like the December into January usage, Opus 4.5, Opus 4.6, it was really important 'cause people spend extended time with the model and then figure it out, oh, actually, I wasn't pushing hard enough, I gotta go further and I gotta rethink what's even possible with this generation.

- Totally, I mean, I don't know.

I feel like there are people internally at every who have been using it who have been like, oh my God, I think I kinda need a new set of skills to use this model.

And I think you can especially see this with people who are maybe more non-technical internally and who are more on the knowledge work side of things where they're like, I don't even know what I would use this for.

And the people who are orchestrating agents are like, holy shit, I feel like there's so many new things I need to learn.

So I'm curious for you, tell us about the difference between your impression when you first tried it and now.

- Yeah, I think your point on adapting workflows is a really good one.

Quite literally workflows, I'll talk about that in a second, but also just in terms of like, how do I like think about usage of the model?

Because at first, the timing was interesting because it kinda coincided with me transitioning from CPO into labs and going really back into builder mode.

And I think it was about a month and a half or two months into that, that we first had, one of these models available internally.

And I sat there and I was like, I feel like a total newbie again, because I feel like the way that I am prompting or even thinking about decomposing a task is really out of date now with this model.

Like it's no longer, and it's even thinking about the time horizon or the sort of like interactivity model I think has to evolve as well.

Like going from, I think early on, would be like, I have an idea for this feature.

Can we start by doing, like absolutely not, right?

Two, great, like let me express more of the intent.

And then just being, I remember like, you know, you know, March, April, be like, wow, on the one shot, it's already incredibly impressive.

But then it also understands the intent around how we're gonna evolve this and understands like the global context as well.

So I think that's been a really interesting evolution till now where, you know, I was talking to somebody this morning where, you know, I think about doing work at a flight and I was like, okay, I can do most of this work remotely.

And I don't even worry that like the Wi-Fi is gonna drop out because I know that if I set up the right, you know, context instructions, like flash loop, you know, I'll see it, it'll see it through.

And I think my last two months have been full of a lot of times where I will, you know, wish Claude a good night, set it up on like a pretty complex task of something of this like model class and wake up to, you know, actually it's usually done by like two in the morning and I guess it just fiddles its thumbs for the next four hours.

But like, really impressive ability to like complete the swing, get itself out of the situation and I was like, okay, well, Mike asked me to do this complex task overnight.

I got stuck because this remote service went down.

I'm gonna write a like scaffolded like backend for it for now so, you know, I'll document that, I'll, you know, go all the way through.

I have a like good mental model of like how far that's gonna get me and then when it comes back online, I'll fix it, I'll keep track of that fact.

It's just like, it is, I think the most impressive thing for me is like you're just being able to like delegate that kind of level of task and just trust that the right thing will happen by the end.

And of course like you'll review the result and there's still like a whole verification thing that we can and should talk about because I think that's an important part of still completing the swing there.

But it's really forced me to rethink like, what is being productive with all these models look like?

And it is much more like we've talked for a while about, you know, like what is it like when these models are more of like a companion or a coworker?

And it really feels like now it's like a teammate that I can delegate like a lot of work to.

- And what is your day-to-day flow like right now?

Because one of the things I noticed is if you just give it a big task and you monologue into it and you just like let it go for a few hours or overnight, it's like the most impressive model that I've ever tried.

But you know, it's so slow and it's so expensive that I feel like I don't wanna use it for day-to-day tasks.

So what is your actual flow like in terms of how you use it day-to-day and where does it slot in versus other models?

- Yeah, I've ended up having a lot more architectural planning conversations up front with it as well.

So that's been like another interesting change where I think there's an area that I think all models need to continue to improve.

And I'm really grateful for the Instagram experience of having to like start, you know, from our initial version that was like duct taped on a server in LA to like being able to scale it and eventually integrate it with like all of like the Facebook infrastructure because you kind of develop a sense of what infra abstractions and complexity are appropriate for each stage of it.

And I still don't always go back and forth with Fable where it'll be like, "Oh, this is a good implementation."

I'm like, "Well, I do plan on shipping this like fairly soon.

Like I think we should probably think about more than one server and kind of like that back and forth is important."

But like a lot of that sort of planning and I'll often actually ask it, it's kind of a thing I've realized is Fable can like be so sort of complete in its thinking in terms of how much you are sort of planning with it.

They often just saying, "Can you just like make an HTML page like that represents what we just talked about so I can share it with the team is actually valuable or even just a markdown document, but I like having diagrams."

So that's been an interesting like use of like, let's plan with it, let's think it through and then let's have some sort of document that we can align the team on because then this is a dynamic I've seen in labs and just teams beyond anthropic which is you can build a lot very quickly and forcing more of that early alignment.

Even if you do an initial prototype and then back it out into more of a sort of plan architecture that works too, I think is really, really, really key.

And as a being ends up being the place for like the human to human interaction still stays very, very much part of the process.

And then from then on, I think, either overnight or during the day, like having it execute on those chunks of tasks is really important.

And it just means having a lot more concurrent sessions than I did before, because I often will think, all right, there's these two pieces of work.

I go back and forth between liking having one like very long running cloud code session and really asking it to do everything in background sort of for sub agents so the main thread stays responsive.

And then other times just embracing like, I'm just gonna, it's one of those days we're gonna have like five or six tabs like tackle like long comprehensive work.

But I do think that there's something to this like long horizon like, don't worry, I'm on it.

It's gonna take me a while and like more of like this back and forth and that modality, I think is something that we'll have to figure out in our products as well.

I think you wanna preserve both and they interact with each other in interesting ways.

And like my preference is usually, I always like having at least one cloud that is high context but also very, very fast response.

And like its instinct is, right, I'm gonna answer you and I'll kick something off if I need to.

And if not, I'm just gonna hang tight and wait for the next kind of loop.

I do think you're right that for the, I'm just trying to fix this interaction question or something that's like very fine detailed.

Like Fable will go off and think very hard about those things.

And I think Fable is the first model where I've actually played more with the effort levels for that reason where I've been like, okay, this is, I just needed to like tweak some UI.

I'm gonna fall but like, no, put it to medium or something and see how that plays out.

Didn't find myself doing that as much with the Opus maybe because the range felt less like wide where it really can feel quite wide with Fable.

- What about like a quick question?

Like you're on the go, like are you asking Fable, you know, random questions as they come to you?

'Cause it feels like you're using a rocket launcher to kill a mosquito or something or are you flipping back and forth?

- It's so funny you asked that because I had been and you know, you're like, it's thinking, it's thinking really hard about it.

Then since last week, like, you know, I was asking it something that like true, I felt embarrassed actually asking Fable about it.

It was something like, probably something NBA finals related.

And I was like, okay, I switched my iOS app to SADA.

I was like, oh yeah, I use this all the time for fast questions.

It's like counter magnitude, like in feeling of like, and it's actually not even the sort of like tokens per second, it's actually probably more around how much thinking goes into the answer.

And sometimes like the answer does not need to be fully thought through.

So yeah, I'm thinking myself through it.

And I think this is a good product question for us too, which is, you know, in general, you don't want people to have to be thinking so much about these choices.

So ideally what we can sort of coalesce around in the longer run is sort of, you know, maybe like some more bucketable use cases that are really grokable to people, or maybe it varies by surface, where it's actually probably unlikely that most of the time with the iOS app, I'm doing Fable type tasks and, you know, having a sticky model selection per surface might be the way to do that.

And we'll have to sort of explore what that means from a product perspective.

But I've for sure, I've had the feeling of like, this is not a Fable worthy question.

I should ask Sonnett.

Can you show us something that you've built with it?

- Yeah, so one of the things that we did this go around is we encouraged personal sort of like account usage for us, like especially on the weekends, which was really fun because we have, you know, you can imagine like a lot of entropic specific, you know, tooling, et cetera, but it was really good to sort of step back and just like, you know, pure cloud code, let's like work on something over the weekend.

- And you're in the terminal app or you're in the desktop app?

- That's a great question.

I'm mostly still in the terminal app.

It's interesting watching my wife who's like not a professional engineer and more of a UX designer PM, like really fall in love with cloud code via the desktop app.

And I think it's like sort of simplified some of the abstractions for her in that way.

But for this one, I was still, it's a ghosty or ghosty, ty ghosty and the terminal app.

But let me show you, I, this is one of those like, and everybody has a bespoke need around this.

Like I wanted a sort of media tracker experience and I was like, you know, I'm playing games, like I'm watching TV, so I get all these recommendations and I just wanted to build something like that was personal to me and like sort of fit some of the use cases that I had.

And I got like the two biggest criteria that I started with was like one, like really easy to add things.

And so like you can talk to Claude, Claude does the Gentic search over everything and then puts the right things in.

And then also proactively, like, you know, there's a new season or a new like sequel to a game that it could go off and research those things.

Most of the UI was like, you know, fable one shot, which was already impressive.

But then the threat I've been pulling out a lot in labs this year is how do you sort of bring the software team, which is Claude these days, closer to the software itself.

And so this was like maybe, you know, Saturday morning, I had a full weekend with kids stuff.

So a lot of this was sort of kickoff work, go do, you know, go for a hike with the kids, come back, you know, continue to do the work.

Sometimes check in on the work on the hike.

I probably shouldn't, but you know, it was like nice to be like pop into the remote code and see what was going on there, you know, try not to do that too much.

But I had this idea around, hey, like, could you really like do a spike on, or I say spike a lot with these models, and be like, can we do a spike on like, what if you could actually modify the software from within itself, which is, you know, and it was, I built both, like a React native version and then this version was just the web version.

So I already had like a chat type thing where you can sort of ask cloud to, you know, add things by URL, which is like, you know, I want every software to have this where I should never have to like navigate a menu to do anything ever again.

And this is like, in many ways, Dan, like the, I was trying to distill the like agent native architectures to like its like fullest degree, which is like, also have the agent be able to modify the app.

So like maybe like phase one of agent native architecture, like every single thing in this product is, you know, accessible from the agent and like that has tool calls, et cetera.

That's like, you know, hopefully becoming table stakes.

It was sadly not in a lot of software and it's great 'cause I was like, what's that like?

Somebody had recommended there's a Brazilian, there's like a show about radioactive stuff going on.

I did not remember what it was called and cloud was able to figure it out.

It was like so much better than being like trying to figure that out intuitively.

But then the next step I was interested in is like, what would it mean to actually be able to modify the software from itself on the go?

And so if you long press this little chat thing, so what it actually, what I built, what cloud built, was a way where it used our managed agents to basically take on like edit requests and then you can preview them.

And then you just like the Resale Live preview thing here.

This like, this whole like feature was also one shot, which was really cool.

- Oh my god.

- And I just added to it over time, but you know, it was like, it actually does like a little diff view if you wanted to.

You can go into the managed agent conversation and see like what it did, although I almost never do because again, it's like, especially don't particularly care on like the current quality of like, or the like long-term maintainability of this software.

You can see that it had a session in here too, but it's been really fun.

So I'll be using it on the go and say like, you know, I had a feature request either day, like, oh, like the floating action button was too low on native iOS, but it was okay on there.

Like, can you go or do it?

It did it.

It was really fun with some of the like expo tooling now and actually like live reloaded on my phone, which was also like a really cool kind of feeling.

But it was just like, you know, does this thing need to be like a, you know, production level thing that's gonna go to a million users?

No, but it felt really good to have something where I felt like it didn't have to stop at just the weekend and I could keep working on it just by using it and having this like kind of end-to-end close thing.

So I felt like this was a good manifestation of both like Fable's building ability, but also like, I think a lot of both of I even think, both you and I have been thinking about like, how does Clod embed itself and like into software beyond just even the usage side of things.

- This is really cool.

And I want people to understand like, so this has been built, you could build something like this, maybe not the self-modifying part, but you could build something like this for like 10 years or 20 years or something like that.

But the cost to build has like gotten dramatically lower.

So think about how much it would have cost to do this in the Instagram days versus now, like, can you help us understand like how that has changed?

- Yeah, I think, and I think about this a lot when I think back to that time as well, because I thought of myself as a very productive programmer in the early Instagram days, I was like really into mobile development and we had like a good clarity of things.

And I think the gap from idea to fully realized version of like some complete product, like you were still looking at, you know, moreish days of kind of my all nighters, which was like my natural state is up till four, sleep until noon, not the new symptom to family life, so I've had to shift, but that was like my building thing.

But yeah, I call it Instagram V1, which, you know, probably more features than this thing did, but not by an order of magnitude, was like five days of all nighters, me working on like the sort of front end and back end and Kevin working on the initial filters to get that out.

And this was also like, you know, like built on already, you know, many years that I've been working on iOS pieces as well.

And then the iteration, you know, I think a lot about what we were gated on after that launch, when things went well was we had all these ideas for where to take it, but we were just trying to keep the site up, or we were just trying to like add the one incremental feature and, you know, hashtags take a week to build, but then there's like all the things that you want to continue doing on it as well.

And so I think it's both that shortening of time, like there's still the time required for the idea and the concept and the iteration.

And then the other piece, which is the, you can then iterate on what you have, and I think a really, I think really fun, but also like very, you know, sort of in the flow kind of way.

And then, you know, if now this is me as a sort of professional software engineer, sort of startup founder, beyond that, if you had that idea, you know, and I saw multiple people go through this, and like, it was like, well, I'll try to find maybe consultancy that will take this on, but like now there's like, it's a really lossy process of like, what I want to, you know.

There won't be a good reason, buddy for it.

And I think that the thing that I think is like the most exciting part about these models getting, not just more autonomous, but again, closing that gap between intent and execution is what I've seen it do to people's ability to build who are not like builders.

And the trajectory of these models has been, you know, if something able, you know, of this general mythos class is like in that class of models and eventually, you know, models of, you know, that are cheaper and more accessible to other folks become available too.

And like, as that process happens, like I just think it is just opening up so many, like I got a king the other day, I get very excited about this stuff, if you can't tell, from somebody internally.

And we had built them an internal tool that kind of combined Fable and like access to some internal MCPs.

And she said like, it is the first time in my life, and she works in recruiting.

And she's like, the first time in life where like, I feel like the thing that's in my head and the thing that exists in the world is now like, they're right next to each other, like I can just do it.

And it was like very, like a meaningful moment to her because prior to that, like, I remember these days, these days were five years ago or four years ago where that person, if they wanted a tool, would have to either make do or try to get an internal tools engineer that probably was overloaded with 50 other, you know, requirements.

But instead now they like are just having the time with their lives building.

And I think that is, I think that's cause for a lot of like hope.

I don't think that human capacity for creativity and what's possible is enormous.

And I think like at our best, we are basically expanding the number of people who can then see that through to something that feels real.

- I totally agree.

But I do think that there's a question in the back of my mind.

And I think it's probably gonna be in the back of the mind is there's some people listening.

So I wanna ask you, given everything you just said, is software engineering over?

- Yeah, I think software engineering is different.

It is like dramatically changed.

And as I probably would have defined it if you had asked me around the Instagram time, like what is software engineering?

I'd probably say like, all right, like thinking through the hard problems and like thinking about an architecture and then like spending a lot of time in, you know, like text mate, I don't know what that can be.

But like, you know, like text editor, you're gonna edit those or Xcode, you know, and-- - Watching Ralph Cass, you know.

- Yeah, exactly, right, exactly.

And understanding the intricacies of Django's like ORM layer and then like 15 bugs after you deploy it.

Like so much of that is radically different and collapsing into other parts of like product manager.

And I think that sort of like PM edge split, I think even in our teams has become much more diffuse.

That's radically changed.

But I think the overall like, like maybe zoom out from software engineering and think about like software production or you know, software development, but not in like just the pure developer case.

I think that is like alive and well and essential still.

So I think that is the moment that I feel like we are in.

I think Fable is another step on the direction of, and I'm not gonna call it the final step.

Of course, a lot will still happen, but like I think a pretty significant step in terms of like the trust, at least I end up placing the model in terms of its capacity to see things through and even architect things reasonably is quite high.

So that part feels like it is not ever gonna be done, but it is pretty done, right?

Like it's gone really far, but I think that the overall sort of craft of what needs you have, like what are you putting out?

Like, is it actually good?

I think still a very human endeavor, but I also sort of can see that that is not a transition that is sort of pain free in a way.

Like I think there are plenty of people who love the craft of like actually putting, I used to love the stuff like I solved that problem so elegantly, you would dream about code and if you were at the experience of like you dream about the thing that you're working on, they like wake up in the morning like, I figured out how to solve this thing really elegantly.

And that for sure has passed.

And I think that there's, you know, there is a feeling of loss I think in some of the like better engineers that I talked to as well as the feeling of, oh my God, but I can do insane amounts of work now at the same time.

So we're holding both ideas in our heads at once, I guess.

- Which I think is the most important part of this.

Like it's normal to feel sadness for that kind of thing.

And excitement, but I'm curious, let's just take the thesis of software engineering is alive and well.

What does that actually look like inside of Anthropic?

- Yeah, I think there's a few pieces.

I think there's still the crafting of, well, I gotta take it off from like the full software development cycle or like maybe what I see on a day to day, maybe I'll do a little bit of both.

But I think there's still a lot of, you know, we all got together, we talked about the next way we want to evolve, co-work.

And now we've kind of broken it down into areas of ownership.

I think that ends up still being quite important because there is still context that you hold as a person that is sort of beyond cloud, right?

Like what is the actual intent of this product?

How's it going?

What do we need to know about the sort of other products that are coming down the pipeline that are going to be integrated in some interesting ways?

So I think that aspect is really important still.

And so, you know, though we have many clouds to each human, each human, at least the way we've been working on anthropics still kind of has, you know, we call them DRIs, like directly responsible individuals, still has like a DRI ship over some part of the product or some area.

I think that'll be the case for a while because I think there is value in not just this distributor, like we should all make co-work better, but instead like, all right, I'm thinking through how co-work does at this particular task.

And there's still a lot of, you know, the try to keep meetings minimal, but they still emerge and you still have these kind of alignment conversations.

Then like a lot of that sort of asynchronous delegation, I think what many engineers here have now found is they've all built, and I think we should solve this at some point at like a broader product level, but they've all built some version of, all right, I'm gonna now like create a dashboard of where all my clouds are doing and what's waiting for me and which floor quests like need my attention because, you know, either a human or a cloud code reviewer got back to me.

So there's a lot of that sort of meta maintenance of the work that I think, again, I think will standardize some, but I think some of it will always be a little bit bespoke to the way each individual likes to work, just in the way that people organize their windows, now they organize their work.

And then there is, I think also the understanding how things work in production.

And I think that is another, like, there's a few like next frontiers, I think, for the models.

And I think one of them that Fable does, you know, make significant strides in, but I think there's more work needed here is understanding what happens to code after it gets deployed, you know, because there's incidents.

There's, you know, this was all working well, but like this network link got cut, which is not in your usual failure mode.

And like it manifested like so much of Instagram, like 2012 to 2016 was like dealing with that and scaling things up.

And so that role of the engineer still remains really key.

And I think getting the reps in around incident response and understanding how to stay calm, gather data, like remediate what's in media, but then like go off and work on longer term fixes, like still a necessary part of it.

And I'm trying to think if there's any like other pieces that are notable as well.

I think what's maybe the last thing to say is I really like the role that the engineering prototype now plays, you have to be clear when it's a prototype versus not, but, you know, the old phrase was like, code wins arguments.

And I never like loved that because like, kind of sort of the person that could code could go do it, but actually like, why should they necessarily win an argument by default?

But actually it's been really cool now where sometimes we will have some disagreement or some sort of debate about where to take a product.

And often it's the PM that will say, all right, I just tried it and like jank in like these eight ways, but look, it actually shows like how this could work and that can open up some interesting pieces of conversation.

So almost all of that is quite different than it was six months ago.

I think especially at the level of parallelism and the level of need for these kind of higher order abstractions of work, but I think what hasn't changed is that ownership.

- Lots of us are shipping AI to production, which is great for productivity, but it also comes with anxiety.

You tweak a prompt, swap models, adjust parameters and everything looks fine in testing.

So you merge.

And then three days later or even sooner, the support tickets start rolling in.

The AI is giving your customers unexpected answers and you have no idea when it happened or why.

Brain Trust is the AI observability platform that fixes this.

It connects evals and observability in one workflow.

That way you see what actually happened in production and can measure whether changes made things better or worse.

Traces show the full execution path.

Evals define what good looks like and experiments that you compare prompts and models side by side before shipping.

Production traces feed directly into your eval data sets.

Every failure becomes a test case.

You catch regressions in CI before they reach users and teams at Notion, Stripe, Zapier, Vercel and Ramp use it to ship quality AI at scale.

Brain Trust is designed for teams building production AI systems where silent regressions are expensive.

It's built for any stack.

They have SDKs for Python, TypeScript, Go, Ruby, C#.

There's no framework lock-in or vendor dependencies.

It's SOC 2, Type 2 certified and GDPR and HIPAA compliant.

Get started at braintrust.dev.

That's braintrust.dev.

And now back to the episode.

Fable is also very expensive.

And because of that, like when I was testing it, I felt kind of like I was a kid in a candy shop and I was just like, I'll do this and I'll do this and I'll do that.

But now that there's gonna be a bill, I'm gonna be thinking about it because I have to pause before I do it to be like, is this gonna cost me a hundred bucks or whatever?

And I do think that's gonna limit who gets to use it and for what.

So how do you think about that?

- Yeah, I think it's most clear cut on the sort of professional software, you know, sort of classic company doing work.

It'll be really interesting.

It's like, you know, a lot of processes that goes into pricing as well.

There's like, it's both more expensive than Opus.

And then also I'm like, in many ways, it's really cheap.

If you think about, you know, like how much incredible work it's doing.

But of course, like everybody has their own economics around what they're working with.

So anyway, most clear cut, I think, from most sort of software teams.

And I think as an industry, if like phase one was, companies even struggling to get some of their employees to adopt AI coding, which models were early, maybe the toy wasn't there.

And then phase two was great, we'll create leaderboards and see who can use the most, which, you know, as you can imagine, creates like some like also like not ideal incentives to phase three, where people were like, okay, now we're just trying to figure out who's using it effectively and like letting them spend as much as possible and having a clear process for that, but making sure we're not doing things wastefully, which I think to me, in general, makes sense.

Although I think you could like also over rotate that way too.

I think something of fable class should hopefully fit in well into that where if you're demonstrating results and you're getting use out of the model, then that hopefully there's a flywheel even inside companies where that goes and perpetuates that.

I think on the personal use side, it's a really good one.

That's a really good question.

I think where I've seen it, you know, even in my personal testing, 'cause our personal accounts, okay, which is funny, like paying my own company I work at, but you do become more thoughtful about it.

Something that was interesting was this, the app that I built over the weekend actually fit in with like only a bit of extra usage.

So it wasn't like, you know, thousands of dollars to build this thing that like is a personal thing to myself, but it was also spaced out a little bit more.

Probably the in-between of that, what we'll probably have to do the most thinking about is the sort of hobbyist or like independent who's like not, you know, within the larger company, but also is thoughtful about the pricing as well.

I think like my overall advice is like, just give it a try and see how much it can do without you having to do a lot of follow-ups.

And it's like, I think the measuring cost has gotten so multifaceted now because there's the per turn costs.

And then there's like, what did it cost you not to just do the task, but like complete the task to your satisfaction?

And I think that's where Fable has really shined for me, which is that actually just does it right so that I don't have to go spend the like 9, 10 subsequent turns be like, no, that was not quite what I meant.

Like, can you also do this piece?

- It's been really impressive for me because you ask it to go do something and then it just does, it does a thing.

And you're like, wow, you thought through all the little details of this thing in a way that I've never seen another model do.

I don't know how much you can reveal about the training process, but what makes the model different?

- I mean, I think in many ways a continuation of a lot of the work that the team has done and I like bow down and total awe of our teams, both on the pre-training and on the RL side.

I think that the piece that it has evolved in that at least I noticed the most is kind of adjacent to that as well, which is a sense of the system more than just the individual piece of the work.

Like I will often be very positively surprised when it will write something and say, all right, but I know that like in production, this needs to be different.

And then it will keep bugging you like, have you turned on that like feature flag yet?

Like it's not gonna work until you do.

And you know, sometimes being sessions that have gone on for days and be like, look, you still haven't done that thing, like you better, like I was like, you're right.

Like I didn't turn on that feature flag.

I should go off and do that.

Or if we change this, the contract will change over there.

Or watching it actually one of my favorite times of seeing it in action, I think where it demonstrates some of the training is watching it respond to code review feedback, either from people or from other cloud reviewers, where it doesn't just say, oh yeah, that's an issue.

I'm gonna go fix it.

And actually really thoughtful around, hey, like for this level of like sort of fidelity of what we're building, I'm gonna accept this risk.

Or I see what you mean other code reviewer, which is often just another fable model like talking to you like, I see what you mean, but like I'm actually gonna push back.

I think that that's actually not right.

And I think getting the model to have that judgment is really important.

And I think if I had to pinpoint like an area where I feel like it's really progressed, it is that sort of not just immediate knee jerk, yeah, yeah, that's right.

I gotta go fix it and more, oh, I'll think about that for a minute.

No, I thought about it and I still disagree.

And I think that's a very useful sort of ability.

It's so valuable to have products like Cloud Code out there because you have now like a living breathing thing where people are like, this is where the model is doing well.

And like, you know, we have like people who tested and count the every folks is like very, very high on the list where like we really trust the feedback because it is being put to spaces and like repeated multi-day you know, hard tasks.

And that also like very much feeds into how we think about like what do we need to improve on the next slide?

Like what are the tasks that we need to specifically think about the model being better at?

- Is chat the right interface for this model?

Because it's not very turn by turn.

It's very like I'm delegating something for you.

So how does that change how you should use it or how you think about the interface?

- I don't think like the fundamental like you are like sending messages and it is giving your message back is like totally wrong.

I think that there's ways we need to evolve it.

Like one is maybe like three that come to mind.

Like one is your laptop the right place for it.

So I think that's number one where I mentioned with the side project I was working on how useful it was to have the mobile side.

Boris who created Cloud Code, he's always like, you know, ahead of the curve on how these models get used.

Almost a year ago, maybe nine months I was talking to him.

He's like, yeah, I've moved a lot of my Cloud Code work to mobile.

I was like, no way.

And like, it took me a while to get there but especially with the FAML class, like there's oftentimes where, you know, because it can keep the session going and we use like kind of remote dev boxes and entropic like it is like a thought like, okay, I need, can you keep up and doing that?

So maybe number one is like decoupling the, the where the work is happening from where I'm talking to about the work.

The second one touches a little bit on what I was mentioning earlier around like, what are, how do you take everything that Fable has sort of discussed or decided or proposed about something and make it comprehensible?

And that's an area that we're thinking a lot about.

Like there are some skills that are out there that we've used around like, all right, can you sort of diagram this?

Can you do that?

So that's a place where the current chat UI, I think is insufficient where like, it will experience this with things, but it will give you like a lot of tech too.

Like this, I need to like take a lot of property to fully understand this.

And I think that that is a piece of property I have something to do with Fable.

It's like, okay, like you have a lot more context on this than I do.

Can you like back it up?

Like, like, let's do like more progressive disclosure of the complexity here.

So I think that that piece is interesting.

The last one that I, you know, I think is we're so early in pulling on is thinking through multiplayer where, you know, at some level, like these, the abstraction levels and like, because we have this sort of DRI and like ownership area, and usually like a chunk of significant work, a human and a couple of clods, like that is still flowing together.

But another case is that is less the case, right?

Where it's, you know, maybe it's an incident response where multiple people are thinking about it.

Maybe it's, you know, a project where there's multiple competing, or not competing, but like conjoining areas that are coming together and thinking through like, what would it mean for, you know, and we have like chat sharing, which gets you a little bit of the way there, but I think there is going to be a need for more like, all right, you've got an independent cloud that's doing a lot of work that was, you know, kicked off by somebody, but can it be keeping up with all the other work happening on the team?

I think that is an interesting and under explored, sort of next frontier about how this work ends up happening.

But I think it's really exciting because I think again, it's the level of teammate collaborator that the models are now capable of, and we're almost holding them back by not having the right abstractions around them for that to happen.

- Yeah, it makes me think, I've mostly been using this for my own vibe coded stuff.

So I haven't really had to, I haven't really had to think about this, but there's a problem when you're using this inside of an organization, which is, do I really understand every part of this?

And therefore, how do I transfer the context of what the model just did into my brain?

Like that's one of the big bottlenecks.

How do you think about drawing the line, especially with a model like this, around how much you actually need to understand and how to make sure that you have enough context on what it's done to feel comfortable?

- I think there's like two big pieces here.

The first is verification, where I became like fully verification code earlier this year, and now like almost in the same way.

And actually it connects to how, I think I used to do when I was typing code more full time, which is try to find the sort of tightest dev loop that you can or on the idea that you're trying to develop in.

Like somebody's with Instagram that meant like, actually making a new build target in Xcode that was just that screen with some sort of synthetic data and just doing that dev loop.

And I would mentor newer engineers, if there's one thing that I can impart on you, like it is try to get that for any project you're working on and things will go much more quickly.

I think that is no longer exactly the case here, but I think what is the case now is, anytime I set it up, like how do I get like for every pull request that cloud is putting out that there is an attached photo or video, whether that's an iOS PR, whether that's something in the UI.

And that's, I think that that helps you gain a lot of confidence because even now, you might have like, you know, fable golf and do work for a couple of hours and be like, I'm done.

And it's really useful to say like, and here's the like full screenshot gallery or the full UI, 'cause you might say like, oh, you know what, on screen strap eight, that error state, I've never actually seen it, but I can see how a person might hit it.

Let's actually make that different.

And so getting that comprehensive verification, I think, it's something we've been working on a lot internally and like sort of publishing more and more skills and knowledge about, but I think is really key piece there.

And then the second one is, I think you ultimately as a person still need to stand behind the work that you are doing, especially if you're putting it into a production system, like a lot of people use cloud every day.

There's still the accountability of like, well, it's still cloud better written, but like you need to understand, you know, the at least the general decisions that were made on these pieces as well.

And so I have seen a fair amount of engineers actually adopt this practice.

We're like, cloud will have done the work, but then there was like the follow up conversation around, well, can I make sure I deeply understand like all the trade offs that you made and that and whatever lowercase A artifacts need to be produced in order to make that comprehensible is important.

- It is really interesting though, to be in meetings where somebody will say like, oh yeah, and I have this PR ready and somebody else asked, they're like, oh, that's interesting.

Like, did you do X or Y and have that moment of positive?

They're like, you know what?

I'm not entirely sure I will find it.

- Not for the poor emerged this PR.

And that's, you know, I think that adapting to that norm and figuring out and work with that is something we'll have to do.

- Tell me more about the verification loop.

It's such a hot topic right now.

Sounds like one way that you do that is with screenshots and screen shares, but what are the other ways that you think about that?

- I think part of it, it starts in, can you get to a place where you are exercising real, like sort of real flows that aren't just like a static injected piece and they think it's more complex, that gets more and more complicated.

So we've invested a bunch into like even just getting it so that the iOS app can log in to staging on a real account and like have real data, but you don't want it to then go through like an eight stage onboarding process every time when you're just trying to test like the second part of the screen.

So there's a lot of work around like, how do you go, you know, is there a special affordance?

Is there like some shared secret, whatever that is around getting the like app, you know, to really feel as human, you know, using the product as possible.

So that's one aspect of it.

The second is like this mix of like well-known paths versus the things you're exercising in the exact moment, like the former being really useful for regression testing.

And so, you know, if I have places where we've expressed like sort of ideal workflows in X basically and the cloud can repeatedly check that.

And then there's also, and cloud does a really good job of this sort of expressing the intent of the current change at hand.

So that gets really, really deeply exercised.

So I think that the combination of those two things is important.

The visual verification that I mentioned as well, video has been really cool to see.

Actually, video is a very underexplored tool to give cloud as well.

Like I think I've been prototyping is just giving cloud video captures of the thing that it has built and then giving it just basically an FFmpeg and you'll watch it scrub through and say like, oh, this animation has some jank in it.

I'm gonna go fix that.

And it never would be able to do it with like a screenshot sort of latency capture because it will have missed the moment.

So I think that's another piece that is, that's really, really important.

And then for the pieces that aren't sort of easily testable intent because there is some more complex system, getting cloud to go and build like as robust as sort of lock back down as possible or use ones off the shelf has been also really interesting.

Like I think when I think about artifact, we had really comprehensive tests.

This is kind of pre-LLM.

And one of the ways that we were able to do that really robustly was that basically every piece of info we had, whether it was Postgres, Redis, you know, all the AWS things had a really good in memory implementation that you could just do really quickly in unit tests and kind of extending that to like cloud land now, you know, working on something where it had like a pretty robust backend and for kind of complicated reasons, hard to spin that up on my dev server, but it was able to again, one shot a really good proxy for that by proxying in like a substitute for that.

And that was so valuable.

And over time, it's been interesting as that like substitute has evolved as the rest of the code is evolved, which is the thing that, you know, if you had pitched that idea to be before, I'd be like, well, that's gonna be really hard 'cause the upstream is gonna change.

How are you gonna keep it in sync?

And I don't think about that anymore.

I'm like, yeah, cloud will read the changes and it'll adapt the thing and it'll keep the two in sync.

And that's fine.

- There's some really interesting architectures around when you get a bug, it just automatically goes out and closes it, you know, the agent just gets kicked off, it closes it, and then it sends a message to the customer being like, it's fixed.

Are you noticing a fable any change in how that process works?

- Yeah, I think there's a couple of things like on a very like human to human or human to cloud level, one of the things that I've seen it do, I've had other models with the cable, I just do it really consistently too, is if the bug report, for example, came from somebody, you know, mentioning something in our like feedback channel and Slack, and then like the thing that got fed into the cloud code session is like, oh, there's this, and 'cause of the Slack MCP, you can actually pull the thread.

Have it then actually post back, you know, as me, and I'll be like, hey, this is Mike's cloud, like I fixed it, here's the, you know, here's the pull request.

But then I think, and the previous cloud's gonna done it, but think of this really well, I would then say, but hold tight, it's not in production yet, I'll follow up when it actually is, and then like maybe a few hours later, like, oh, like this deploy went out, like, you should go test it, is it fixed now?

And like that level of follow through I think is new on the closing the loop piece, and it's five, I don't know if it has these long run and cloud code sessions that are basically like interacting as me, I guess, but put some disclaimer in there too.

And the second goes back to that like taste and discernment piece that we were talking about, which is like, it's one thing to say, there was a bug report, therefore I must go fix this thing, and it's another one to say, you know what, like this, like I hit this over the weekend, one of our internal systems basically had been running without restarting for a while, there was a memory leak, and it was good to start missing like, all right, Mike, like it's the weekend, like just re-bounce the server, it's gonna solve it for now, and like we'll work on the like, I'll synchronously get the PR going to like fix this more long term.

So I think if you're gonna have cloud in the loop in this kind of like, sort of close the loop bug report or system sort of issue to change, I think you really wanted to understand where, you know, as any good SRE or engineer in the loop would like, great, they'll solve the problem at hand, let's like defer the question of like, do we need a re-architect on top of a completely different language found, and understanding that balance is really important.

- One of the things that's like really exciting, mostly exciting to me about new models is, it raises the floor so that everyone can kind of go build apps in one shot, but it also raises the ceiling for experts.

So like if you're a software engineer or a founder, you can just go do things that you never would have been able to before because you have access to this really powerful model.

So for me, I built this one shot version of Borges' infinite library, like a 3D game version of the library, it's wild.

It runs right in the browser.

It's so good, I can find like any, every essay inside of it.

I'll send you the link, it's sick.

But I think there's gonna be this flowering of people doing things like, oh, I made a game, or maybe I trained a new model or whatever that they couldn't do before.

And I'd love to give people some inspiration, some examples of things that they might be able to do that they might not be thinking to do with this model.

What are some ideas that come to you?

- Yeah, I think a few.

Maybe I'll start with the fun side and like riffing off the game piece.

Like I think people have a lot of like creative ideas for how to express the complexity of what they are, like their world, like everybody has the thing that they know really, really well.

And there's probably some level of like, how do I then explain that to somebody else?

Or how do I apply techniques elsewhere that I could then go off and do?

My wife is studying like environmental engineering, like studying geothermal, like very complex math and simulations.

And I've seen like as the models have gotten better, she has been able to apply even more complex techniques from even outside of that domain into that work.

And I think what people should be able to do, you know, like full on PyTorch end-to-end simulations of that work in a way that wouldn't be possible.

I think that maybe is one is like, bring the like beautiful complexity of what you have and either show it to other people by like maybe making a game or maybe making a visualization, which I've seen her do as well, or at least like make, you know, bring other techniques to bear.

And the second piece is, it's ability to compose software that like solves a really unique problem to you.

And I've seen that internally.

A lot of the work that we've been doing is, how do we get as many of our internal systems like MCPF5 with the right permissioning structure and the right deployment kind of set up, although externally you have good options around some of these like platform as a service pieces and you can just ask cloud about them and they'll like help you set things up.

But like, I love that feeling of like that thing that you always wish that you had.

And then what has blown my mind, there was a person who works in our good market organization has been like building this like really, like for deeply thought integration of cloud into every part of her whole process.

And you don't have to stop at that one shot.

Like she's been working on it for months now and she can keep going.

And like, I think one of the things that is maybe underappreciate about models is, I think in previous generations, they would eventually get to a complexity level where it was hard to iterate on it without feeling like you then would break the thing that they had, you know, like under or over abstracted.

Whereas this is actually, you know, she's had access to from the fable or fable light for a couple of months and like, you've just seen it keep growing and growing and growing and growing.

And actually is like deploying it to the whole GTM org.

And like, I think that is really cool.

Like the ceiling of complexity that a person that does not start out as technical can now build for solving problems within their domain is like, is unprecedented.

- I agree.

It writes great code.

Like my benchmark that I have is called the senior engineer benchmark.

I just have it see if it can rewrite code base from first principles and the nearest model, the previous top was like a 62 or 63 out of 100.

And this model got a 90 on the benchmark or 91, which is human senior engineer level.

Like you can just keep going with this thing in a way that's, it's really fantastic.

I'm curious though, one of the things that's really powerful that you mentioned is dynamic workflows.

Tell us about that.

- This is, we'll build things internally sometimes and I will go really aggressively, bug the engineer who built it and be like, when are we shipping this publicly?

'Cause I think people are gonna really like it.

It was a nice, there's good reason why it was like built internally, but like we try to ship as many of these as possible.

And dynamic workflows was like definitely that to me.

I personally built this as an engineer name, Sid who's awesome.

And it was like, Sid like, I want to get this out to the world 'cause it's so good.

But I think it's especially good with a model like Fable for two really big reasons.

One, it helps sort of create the scaffold for like deep meaningful work.

The craziest dynamic workflow I did and used Fable for was I had an internal project that we had written in Python, but we needed it actually in TypeScript for like a really specific deployment reason.

And having been internal to Instagram, we were like, should we write the whole thing into hack and you know, port it to the PHP engine that Facebook?

I was like, you never would have done that.

Like maybe they can now with the model.

But you know, at the time it seemed impossible, but here I had, you know, pretty complex code base.

And I was like, I'm just gonna set up a dynamic workflow and just let it run over the weekend.

And it did, and the workflow was so cool.

It was like, all right, I'm gonna do like a deep understanding of the work.

I'm gonna create sort of like almost like a spec of how everything works.

I'm gonna go module by module.

I'm gonna translate these pieces.

I'm gonna have tested incrementally.

I'm gonna do another adversarial test.

I'm gonna go check for anything that I missed.

And it was just like really cool like series of steps that the workflow was able to orchestrate.

And I came back and I was like, yeah, this thing is like TypeScript and Bund port of that thing.

And it's actually better in these ways.

And it was very sort of documented.

Like these were things I couldn't port, but most of these were like very specific to this specific implementation.

It wasn't worth porting.

And I do not think you could have done that, A, with previous models at that level of success, and B, without like the kind of scaffolding that Ork Flows provides.

So I think that is extremely exciting kind of combination of model capabilities and then our own ability to like orchestrate them over a longer and longer time horizon with that feeling of like, you had a goal, you broke it down effectively and then you were able to make it work.

The other piece is I think over time, we'll be able to also make some of those sub tasks sort of have the model be tuned to the level of complexity of it.

So you can imagine that some parts of dynamic workflow don't need extra high thinking.

They could use a medium thinking to get it done or even a smaller model.

And I think that's really the future where these things are going.

So yeah, I'm a huge workflows DAU.

- For people who haven't used it before, tell me about how you got that workflow made.

How did you design it?

How did you make sure it was good?

- Yeah, it was pretty iterative, but sort of just started with Cloud Code.

Like, hey, I have this complex kind of task, like let's design a workflow to go into it.

It kind of showed me the plan.

I was like, oh, this is like close to what I want.

I want to make sure that you do these three or four levels of additional verification for missed features.

Here's what you have.

You're ready to go.

And it expressed the workflow's encoder, which I think is really valuable to kind of see what it was about to do.

And then what was interesting is it did the full port.

And then I had like a couple of like follow up kind of questions that I had or like little tweaks.

I did those as sort of like mini workflows that built off the previous one as well.

But I think that's like, you know, we talked a little bit about whether chat was the right.

We've had that conversation over the last year.

And I think workflows are a good middle ground of you can compose them using chat, but they're expressed using code.

And then they are executed with like, I think a nice clean UI around what's happening at every stage.

And like, I think we'll start bridging longer horizon work with chat in ways like that over time.

- Mike, this is such a great conversation.

Thank you so much for joining and telling us all about this new model.

- I'm really excited to just spend time with you and really, really look forward to what people think outside too.

(upbeat music) (upbeat music)