OpenAI's Sherwin Wu on AI's Transformative Impact on Software Engineering and Beyond

Source material

Transcript

95% of engineers use codex.

100% of our PR's are reviewed by codex.

Four engineers.

I don't know what job has changed more in the past couple years.

Engineers are becoming tech leads.

They're managing fleets and fleets of agents.

It literally feels like we're wizards casting all these spells.

And these spells are kind of like going on doing things for you.

What do you think people aren't pricing in yet?

The second are third order effects of the one person billion dollar start up.

Two enable a one person billion dollar start up.

There might be a hundred other small start up spilling the spokes off for it.

So I think we might actually enter into a golden age of B2B SaaS.

I've been hearing more and more.

There's this stress.

People feel when their agents aren't working.

There's a team that's actually doing an experiment right now with an open AI where they are maintaining a 100% codex written code base.

They run into the exact problems that you're describing.

And so usually you're like, all right, I'll roll up my sleeves and figure it out.

It doesn't have that escape patch.

You've shared that listening to customers not always the right strategy in AI.

The field and the models themselves are just changing so, so quickly.

They tend to like disrupt themselves.

The models will eat your scaffolding for breakfast, which are advice to folks that are like, oh, I don't want to miss the boat.

Make sure you're building for where the models are going and not where they are today.

But there's a quote from Kevin Whale, our VP of Science here.

And he likes saying this is the worst the models will ever be.

Today, my guest is Sherwin Wu, head of engineering for Open AI's API and developer platform.

Considering that essentially every AI start up integrates with Open AI's APIs, Sherwin has an incredibly unique and broad view into what is going on and where things are heading.

Let's get into it after a short word from our wonderful sponsors.

Today's episode is brought to you by DX, the developer intelligence platform designed by leading researchers.

To thrive in the AI era, organizations need to adapt quickly.

But many organization leaders struggle to answer pressing questions like, which tools are working, how are they being used, what's actually driving value?

DX provides the data and insights that leaders need to navigate this shift.

With DX companies like Dropbox, booking.com, Adian, and Intercom, get a deep understanding of how AI is providing value to their developers and what impact AI is having an engineering productivity.

To learn more, visit DX's website at getdx.com slash Lenny.

Let's get DX.com slash Lenny.

Applications break in all kinds of ways, crashes, slowdowns, regressions, and the stuff that you only see once real users show up.

Century catches it all.

See what happened where and why, down to the commit that introduced the air, the developer who shipped it, and the exact line of code all in one connected view.

I've definitely tried the five tabs and slack thread approach to debugging.

This is better.

Century shows you how the request moved, what ran, what slowed down, and what user saw.

Sear, centuries AI debugging agent takes it from there.

It uses all of that Century context to tell you the root cause, suggest a fix, and even opens a PR for you.

It also reviews your PRs and flags any breaking changes with fixes ready to go.

Try Century and see your four free at centri.io slash Lenny and use code Lenny for $100 in Century credits.

That's s-e-m-t-r-y.io slash Lenny.

Sure, when thank you so much for being here and welcome to the podcast.

Thank you, thank you for having me.

I want to start with what's feeling like a barometer of progress in AI, especially in engineering.

What percentage of your code, if you even write code anymore and your team's code, is written by AI at this point?

I do write code occasionally now still.

I actually say for managers like myself, it's way easier to use these AI tools.

Then to manually code at this point.

And so I know for myself and some of the other EMs, engineering managers that open AI, all of our code is written by code X at this point.

But more broadly, there's just been this, there's just so much energy.

There's like a tangible energy internally around just how far these tools have gotten, how good code X as it's well as gotten for us.

And it's a little hard for us to exactly measure how much of the code is written.

Because the vast majority of it, I say a close to 100% is usually generated by AI first.

What we do track, though, is at this point, the vast majority of engineers use code X on a daily basis.

So 95% of engineers use code X.

100% of our PRs are reviewed by code X.

Daily is also basically any code that goes in a production that's merged in.

Code has kind of has its eyes on and suggests improvements, suggests changes in the PRs.

And so that's kind of what we're seeing internally.

But by and large, the most exciting is just the energy that there is.

Another observation that we've had is engineers who tend to use code X more open way more PRs.

So they're actually opening 70% more PRs and then the engineers who aren't using code X as much.

And the gap is widening.

So I feel like the people who are opening more PRs are certain to learn how to use the tool more.

More, get more efficient and that 70% gap keeps growing over time.

And so might have actually increased since I last looked at the number.

Okay, so just to make sure we hear what you're saying.

You're saying all of the code of these 95% engineers at OpenAI is written by AI.

It's written and then they review it.

Yep, yep.

It's like crazy that that's almost like not crazy anymore that we're just like getting used to this.

I think there's still some getting used to to be clear.

There's also I think some, you know, engineers who I think trust code X a little bit less.

But basically every day I talked to someone who is blown away by something that I can do.

And kind of like the their bar of trust kind of or like how much they trust the model to do on its own goes up over and over over time.

And there's a quote from Kevin Whale are a VP of science here.

And you like saying this is the worst the models will ever be.

And so this is the worst that the models will ever be for software engineering as well.

And so over time we just see people trusting it more and more.

And then we'll see the models get better and better as well.

Yeah, Kevin Whale former podcast guest, he he said exactly that line on this podcast.

Yeah, yeah, yeah, a few times.

Yeah, Peter, the club bought slash mulled bot slash open clause.

What it's called now developer recently shared that he uses code X for his work.

And he feels like anytime it does them he just trusts that it has done the right job.

But he's just like almost certain he could just commit it to master and it'll be great.

Yeah, yeah, he's a great user of code X.

I know he's in close touch with the team is a great feedback.

I'm not surprised that he uses it.

I mean, sorry, it's called open clause.

Yeah, it's a classic great is a great product.

And then I saw that just more.

I mean, this is very recent that this morning.

I think the most book kind of like a with shared as long seeing all of the AI agents talk to each other.

It's pretty surreal.

It's basically hers happening in real life as well.

Yeah, yeah.

So just like coming back to this crazy moment we are living through four engineers in particular.

We've gone from you write every line of code to now.

AI is writing all of your code.

I don't know what job has changed more in the past couple of years.

Like job that we didn't expect to change this much.

We're just like the job of an engineer is so different in the entire life span of an engineer.

Like in the past couple of years, it's now shifted to I don't write any more code.

How do you imagine the role of an engineer in the job of a software engineer looks in the next couple of years?

Just like what does that job?

Yeah, it's I mean, it's honestly been really cool to see.

And it's part of where the excitement is because like the job is likely to change pretty significantly over the next one of two years.

It kind of feels like we're still figuring things out though.

And so there's like this excitement.

I know especially from some of the software engineers of like we're in this rare moment.

You know, maybe over the next 12 to 24 months where we'll kind of get to figure things out ourselves and set our standards for ourselves.

In terms of where I see I see this moving.

So I think there's a commenting that everyone's saying, which is, you know, people are generally like I see engineers will be coming tech leads.

They're basically like managers now.

They're managing fleets and fleets of agents.

I know many of the engineers on my team basically have like 10 to 20 threads kind of being pulled on at the same time.

Obviously not active running codex jobs, but just a lot of parallel threads they're checking in on what they're doing.

They're steering the agents and codex and and and giving it to the back.

And so their job is kind of really changed from just writing the code itself into being almost like a manager.

In terms of where I think this will go one to two years now.

So one kind of metaphor that I kind of always come back to here is actually from this from this programming textbook that I write back in college called sick be.

I don't know if you've heard of it.

structure and interpretation of computer programs so S I S I C P.

At MIT it was really popular and and it was actually used as the introductory it was the textbook for the intro programming course for a very long time.

And it kind of has this called following it teaches you programming it teaches you a dialect of list called scheme and so he like introduces you to like functional program is like very mind mind opening that way.

But the thing that was memorable for me about that book so I I kind of read it in college.

The very beginning of it kind of describes programming as a discipline and draws this metaphor to basically like sorcery like it says like software engineers are like wizards and you're like you like program language or like incantations and you're like.

You know you're you're saying you're issuing these spells and these spells are kind of like going on doing things for you and and the challenges like what incantation you have to say to make the program do what you want.

And this book is written in 1980 so this is a while ago and I think that metaphor is actually like kind of persisted over time and I think it's actually playing out as we move into this new era of vibe coding or just like what software engineering will look like because programming languages were basically is incantations they've changed over time and the challenges always and and the trend has been that these you've been easier and easier to kind of get them the the computer to do what you want to be a programming.

And I think the current wave of AI is is probably the next stage of that evolution it is now literally incantations because you can tell you know you're you can tell codex can tell cursor exactly what you want to do and then it'll go do it for you.

And I particularly like the wizard and like the the source analogy because I think our current status is starting to move towards kind of like the the sorcerers apprentice you know from fantasia we're making houses like you know he finds the sorcerers had any tries to do all these things and.

I think it's a really apt analogy because one is just it's really powerful now these incantations you can do can is extremely high leverage.

But you kind of have to know what you're doing right like in sorcerers apprentice the whole plot is like Mickey goes wild the the rooms like go crazy and everything's flooding I think he literally sets the like sets the the rooms off on a task and then goes asleep and and so you know it's like vibrating at its at its greatest.

And then eventually the the the old sorcerer comes back and like cleans everything up and you know when when I see engineers kind of like doing these these these these 20 different codex threads at a time there there's some skill and there's some seniority and like you know a lot of thought that needs to go into this because you want to make sure that the the models aren't going off the rails.

You definitely don't want to just like completely go away and and you know like ignore ignore the thing but it's also extremely high leverage like you know a very senior engineer who's who's really prolific.

proficient with these tools can now just do way more things the what they're doing and I think it's also what makes it fun like it literally feels like we're wizards not you know feels like we're closer to to to having.

To making making it feel like it's like magical experience where we're you know casting all these spells and having software do all these things for you.

I was thinking of the sorcerers apprentice exactly as the metaphor as you were describing that some glad you went there.

A previous podcast guest described as you have a genie that you can that grants you wishes and it's a useful frame because you have to be very clear about the wish you want.

Like if you want to be big yes, I'll take it.

Yeah, or might be like the monkey's pod type thing where you know it's actually caught what you want but what are the effects.

Yeah, yeah, I think that the analogy is great and yeah, the crazy thing for me is just the staying power of that book sick be like it's called the wizard book you know people call the wizard book because that is the metaphor that they kind of weave throughout the the book and.

We're we basically reach that point now, which is which is really cool.

There's two kind of threads on a fall here.

What is I've been hearing more and more.

There's this like stress that people feel when their agents aren't working.

Do you fire off all these you know codex agents and then you have to keep stand out of them oh shit.

What's that working and wasting time do you feel that you feel that across your team at all.

Yeah, yeah, I mean it happens all the time and I actually think like this is where the interesting part of all of this lies right now because these models aren't perfect these tools aren't perfect.

And we're still trying to figure out how to best interact with these with with with codex or with these AI agents to to get work done.

We see this come up all the time.

There's a particularly interesting team that we have internally.

So there's a team that that's actually doing an experiment right now with an open AI where they are basically maintaining a 100% codex written code base.

So you know like you know something you know you'll have the AI right code, but you'll obviously end up like rewriting a lot of it and you might need to like double track and change things.

But this team is just fully codex-pilled and just like leaning in entirely and they run into the exact problems that you're describing which is like you know their challenges you know I want to get this thing is feature built but I can't get the agent to do it.

And so usually it was going to escape hatch where you know then you're like all right I'll roll it my sleeves I'm like figure it out and then instead of using codex I might use like tap complete and cursor and and things like that.

But this team for the experiment this team doesn't have that escape hatch and so then the challenge like how do I get the the the agent to do this and I actually think we're going to be publishing a blog post from some of our learnings here but a lot of fascinating like paradigms and best practices are falling out of this.

One interesting thing that we've noticed I don't know if this is what you you kind of feel but we definitely feel it here is a lot of the time when the coding aid is not doing what you want it's usually problem with context and just like information that you've given it it's just you've either on their specified or there's just not enough information around how to do something available to the agent.

Be able to codex and so when when you have to solve it through through that the challenge is then to to add documentation and actually work around this this limitation and basically encode more tribal knowledge as in your head somehow into the codebase either via you know code comment itself or code structure itself or via text files like you know dot md files skills any type of additional resources within the repository so that the model can.

can better do its task there's a whole bunch of other learnings from this this group which I think is fascinating to to explore but yeah kind of giving removing that escape hatch of no longer using the eye has allowed them to start piecing the other a lot of the problems that we'll have to solve if you really want to lean in to agents.

Another issue people run into you talking about how people are shipping PRs like crazy a lot more PRs if they're working with a eye.

Obviously code reviews becoming a bigger challenge is there anything you figured out and you're team to help speed that up to make that scale and not just create this terrible job for people or they're just sitting there reviewing PRs all day.

Yeah, I mean one thing is code extra views 100% of all of our PRs at this point and so I actually think so one one really interesting thing that's happened is the things that tend to we hand we tend to hand to the models immediately tend to be the things that annoy us or like the most boring parts of software engineering.

It's also why it's more fun now because we get to do more you know more of the fun things for me speaking more for myself.

I really hated code reviews it was like one of the worst things for me and then I remember I went my first job at a college it was that it was that core I owned I was working on the newsfeed.

And so I owned the code for the newsfeed and so I was a reviewer for newsfeed and it was just like the central piece of code that everyone would touch and so I would just every morning I'd log in and be like.

Like 20 to 30 code reviews and just like oh my goodness I got to like you know get through all of these.

I would procrastinate and then it grows to like 50 and so there's just like a lot of code reviews.

Codex is really good at reviewing code so actually one thing that we've noticed that five two in particular has gotten extremely strongly adopted is reviewing code and especially when you kind of steer it in the right direction.

And so for code reviews yeah we create a lot of PRs but codeics reviews all of them.

And it makes you know code reviews go from a you know I don't know 10 15 minute tasks to sometimes even just like I've two to three minute tasks because you have a a bunch of suggestions already already baked in.

A lot of the times people will especially for small PRs like you you actually don't even need people to review we kind of trust codex in this way.

The original author kind of website codex it is you know the benefit of code reviews after second pair of eyes to make sure that you're not doing anything done codex is a pretty smart second pair of eyes at this point and so that's something that we've heavily lean into.

The general CI process and like the post kind of push and like deployment processes also been heavily automated via codex internally at this point.

If you talk to a lot of engineers that thing that annoys me most is after you've written your beautiful code like how do you get it into production you know you got to.

You have run through all these tests yet like you know limp errors yeah a lot of code review.

There's a lot of automated stuff you can do with codex and so we've actually built some tools internally that that help automate that process automate the lint you know if there's like a link to error it's a very easy codex fix.

And then just it could just patch it and kind of restart the CI process.

So all of that is we're trying to collapse as into as as little work for an engineer as possible which and then the by product of which is they can they can now merge and push out a lot more peers.

Codex writing the code codex for being in some code.

I'm curious if you're open to using other models to review your models work is that is that a path or is it just it's good enough we don't need anything else.

So I will say there's there's only circular thing here and like going back to sources apprentice like you want to make sure you're not letting the rooms go crazy here.

And so you know we were very thoughtful I'd say around which PRs kind of are completely just codex reviewed most people still obviously take a look at their PRs and so it's not like it's going to zero it's more like going from you know 100% attention to like 30% attention which which just helps things push through.

In terms of like multiple models so we obviously test a lot of models internally and so we have a lot of those.

We use external models less it's we think it's important to kind of dog food our own models and kind of like get feedback there.

But you can also you know there are a lot of like internal variants of models that you can use to give you different perspectives here as well and we found that to work quite well.

Okay, so just to just to just to make sure we get a like a barometer of today's world at open AI in terms of AI and code just so understand and then I want to move on to different topic.

A 100% of code across open AI is written by codex at this point is that the way to frame it I wouldn't make a statement that 100% of code running a production today was it is written by AI and it's kind of hard to to do after be from there.

But it's the like almost every engineer heavily uses codex in all of their tasks at this point and so I you know if I were to guess to make it like the vast majority of code at this point is was probably author by an incredible.

Okay, so there's a lot of talk and we've been talking about kind of the icy role the work of an icy engineer.

There's let's talk about the changing role of a manager, especially engineering manager.

How is your life as a manager changed with the rise of AI and just what do you where do you think managers what's the role of a manager in the future.

Is that only changed less than an engineer.

There's no you know codex for managers just however I use codex quite a bit for some of the.

Some of some of the kind of more managery tasks that I do.

I'd say a couple things are are changing there like some trends so I don't think it's changed that much yets.

But I see trends and I think if you play it out you can kind of see where a lot of this is going.

One thing that that becoming increasingly clear is codex really empowers like top performers to to get a lot like to be a lot more productive and so it really like.

And I think this is maybe true for AI more broadly like a cross society which is like the people who really lean in are like the people who.

Who have high agency or like will really get get get get good at these tools will kind of super charged themselves and so I'm kind of noticing this now as well which is like the top performers kind of.

End up being a lot more a lot more productive and so you see a broader spread and team productivity in this way.

One so one thing that I've always done as a management philosophy is to spend actually the majority of my time with top performers just like make sure they're on block make sure they're happy make sure.

You know they're they feel productive and they feel heard I think this is even more true in an AI world where you know your top performers are going to just like really be shooting ahead using these tools.

I think one example is the team that's you know maintaining a 100% codex generated codebase like just letting them kind of rip and and see what's happening there is something that that's paid dividends.

So I think that that's kind of one one trend that I'm seeing where you where.

Spending even more time at top performers for managers I think is is likely going to continue.

The other thing is I so this is more an observation but.

My sense is with a lot of these AI tools available to managers so less like writing code but just things like catchy BT with organizational knowledge like being able to do research and understanding organizational context lot better.

Another good example is we're doing performance reviews right now and it's actually really easy to use chat GPT with internal knowledge hooked up to get hub and like our notion docs and Google docs to give it get a really get sense of what this person has done over the last 12 12 months and writing a little you know deep research report for it.

My sense is I think managers will be able to manage much larger teams and that's what kind of like how you know like software engineers are managing 20 to 30 codex's.

My sense of these tools will allow managers people managers to be higher leverage and will allow them to to manage you know teams of way more than than the current best practice of I think it's like six to eight right for software engineering.

You kind of see this applied to you know like the non engineering domains like support or operations where it's like you know previously.

I were we're previously like the size of support team might be limited but like as you can pass up more things to agents you can actually do more work and also manage more people this way.

I think the same thing might happen for people management as well especially in tech companies and we're already seeing this there's some teams.

Where they're EM's managing you know quite a few people and they're doing it pretty ugly because of some of these tools where they can get higher leverage and understand what their teams doing understand organizational context a little bit better and operate in that way.

I love this advice that with the way you described as you've always leaned into top performers and spent more time at them and block them and make sure they're happy.

The way Mark Andrews and he's just not podcast the way he phrased it is it makes good people better and it makes great people exceptional.

Yeah, and what you're saying here is just just doing this more and more is probably the right move spending more time with the best people on your team to unlock them make sure they have everything we need.

Yeah, a very good example right now is there are I would say like a group of engineers internally who are really codex fields and are thinking through what the best practices are for interacting with this model and that is just an extremely high leverage thing for them to do.

And so just like as a manager I'm just like yeah go explore this you know whatever best practices come out of this you know we we have to share with the org well we'll you know we'll we'll we do all these knowledge sharing sessions will we'll like share documents and like best practices everywhere so things like that just you know elevate everyone.

And and so I view that as like you know another example of this trend that that we're seeing where the top farmers really get exceptional people just like have a sense this is big as changing so much the world is changing.

It's going to be a huge deal.

What do you think people aren't pricing in yet into what will change into where things are heading just like what's an example something you think are like okay we're not realizing this yet.

So well one of my favorite kind of like phrases or like things that have come out of this holy eye wave is is the idea of the one person billion dollar start up I think I should examine if.

I like I'm saying maybe in the first one to say it but it's fascinating to think about right it's like yeah if you know people are so high leverage at some point there will likely be a one person billion dollar start up.

And while I think that's really really cool I think people aren't really pricing in the second or third order effects of this and really what you know.

Because because what the one person billion our start start up implies is that there's you know one person can just have so much more agency and so much more leverage using one of these tools.

That it is just super easy for them to get everything done that they need to for for their business to you know ultimately create something that's a billion dollars but I think there are a couple other implications of this so one of them is.

If it's easy for a person to create a one person bill or if it's also for a person to create a one person billion dollar start up.

It also means it's way easier for people to create startups in general like I actually think this whole like one second order effect of this is I think there's going to be a huge like startup boom and like small like SMB style boom where anyone can build software for anything right like.

One you're just kind of starting to see starting to see this play out in the AI startup scene where.

Software's became a lot more vertical oriented where like these verticals like creating some AI tool for some vertical tends to work quite well because you know you really lean into that particular domain you like really understand the use case for it.

And so if you play out AI there's no reason why you can't have like hundred X more of these these startups.

I think I think one world that we might end up seeing happen is in order to enable a one person billion dollar startup there might be like a hundred other small startups building bespoke software that work extremely well.

To support other types of you know small small one person you know billion dollar startups.

And so I think we might actually enter into a golden age of like B2B SaaS and just like software and serves in general and so I think I think that's that's a really interesting trend to kind of see because.

As it's as it's really as it gets easier and easier to build software as it's easier and easier to you know run a company you might actually just end up seeing way more of these these these startups.

So the way I I've been thinking about is like yeah there might be one one person billion dollar startup for there might be like a hundred you know a hundred million dollar startups there might be tens of thousands of 10 million dollar startups.

And as an individual is actually pretty great to have a 10 million dollars is like that's like enough for yourself or like at that point and so you know we might really see seen explosion.

And that way and I feel like people aren't really you know pressing that in.

There's another kind of like third order effect of this you know and again all of these like as you get to the further and further out predictions I think there's a lot of uncertainty.

I think if we end up moving to this world where you end up with these like kind of micro companies building software that works for one or two people.

Who own the company and are working there.

I think the startup ecosystem will change I think the VC ecosystem will change you know might we might end up in a world where there's just like a handful of big players that are offering platforms and supporting all of these startups but.

You know the types of ventures scale return startups that can really a hundred or a thousand next year your investment might actually an opportunity.

If you end up having a bunch of these you know smaller 10 to 50 million dollar companies which are not great for venture solar turns but are great for the individuals the high agency individuals who are now you know really lean into AI to build these businesses for themselves.

I love how many order like order effects we've been through whenever you have the fourth order effect now sure I'm just joking.

I can I it's to fourth order is two two is to give a brain for me I can't I can't think that far ahead.

It's like conception or just everything it's slower every time you go deeper into some.

Yeah, every layer.

Okay, so the billion dollar startup I've been I think about this a lot because I I'm not going to be a billion dollar startup because what I'm doing is not venture skill in any way and not super high leverage but.

Just could see how many support tickets I get from just like the most ridiculous things.

It's hard for me to imagine one person like I'm bearish on this billion dollar startup I just want to share this thought simply because of the support costs even if AI is helping you.

At a billion dollars just like unless your ACVs are you know very high and you have very few customers.

I just dealing with support and people are like you know like they can solve their own problems with their all email support ask about this thing.

Just dealing with that is hard to scale is in my experience so unless you have in my opinion unless you have a bunch of contractors which I don't know is that count is a single person.

Company I feel like it's very difficult to scale a billion dollar startup and not have someone helping you with at least the support work and I think we'll need to take you so far.

So I think that's true and actually I think my view on it is is slightly different which is I think that your you know, Lenny's podcast might end up becoming a billion dollar startup but what I think might happen is instead of you kind of being the one person who has to dispatch an AI to solve and fix those support tickets.

I think what might end up happening is there might be a whole smattering of other startups that are building software and super and like super tailor towards what you might need.

And so you know there might be like 10 or 20 startups that build support software for podcasts and newsletters and that might be a one person startup like it doesn't need to be a big one and it's and you know they might be able to just code up this product very very easily.

They are able to kind of build their own thing and because it's so tailored and unique and hopefully you know useful for you and might be something that you purchase.

As the one person billion dollar startup.

Yeah, there's like a question of like what you and how some of you what you like kind of outsource and what I think might happen is because the cost of writing software and building products is is collapsing so much.

You might end up outsourcing a lot of this and in doing so reducing the size of your company.

And so that's kind of the world that I think might end up happening again.

I think I answer a new one might play out here, but the end result still might be a one like one person driving this like high high massive leverage company that might actually reach a billion dollars.

I could see that.

I also think about Peter Claude bought slash mode box slash open claw.

I've just like how he barrage these right now by all these asks and emails and things and DMs and PRs just like.

I'm curious to and he's not even making money out this thing.

I can imagine what it's like to be him right now, it's it must be like absolutely insanity.

It's probably like, you know, like the the months after we launched shatchy BT the craziness that was.

Yeah, as one as one.

Yeah.

He's coming on the plot by the way.

And a week.

Oh, I'm excited.

Yeah.

Maybe the fourth order effect is distribution becomes increasingly important because there are so many freaking things trying to get your attention.

So people with an audience and platform, I think become more and more valuable, which is good.

Good stuff.

Okay.

I wanted to come back actually to your management stuff.

So I really love your insight about spending more time with our performers has been really successful to you.

Just thinking about you as a manager of a team that is building the platform that powers basically the entire AI economy.

Like every AI startup is building on your API.

Clearly you're doing it great job.

What other kind of core management lessons have you learned?

What do you find is really important and and key to your success as a manager of engineers and just people?

Yeah.

I think a lot of the lessons that I've learned here.

I don't know how specific it is to the opening API or some of our enterprise products in particular.

I think my my management philosophy is obviously changed over time, but I think it's.

Well, I stayed the same more than it's changed over time.

What one of these principles is is kind of what I talked to you about before, which is, you know, spending a lot of time with with top performers like actually spending and like to be very concrete like it's like more than 50% of your time with your top performers with maybe your top like 10% performers and really really trying your best and power them.

The way that I think about it is is is is kind of come back to this analogy of software engineer as as as a surgeon, which comes from the mythical man monthbook.

So it's actually it's funny.

So I pull it from the book, but in the book they actually describe this world where I think they were like predicting the future because because I think the book was written like in the 70s or something.

They said that software engineering might end up moving into a world where that software engineers are like surgeons or like in a surgery room.

There's like one person doing the work.

And, you know, there's the one person like cutting whatever and like doing all the surgery.

And everyone else in the room is there to just support them, right?

It's like the nurse and like the assistant, the resident and the fellow.

And then the surgeons like I need a scalpel and they give them scalpel and then they're like I need, you know, this tool and that's machine and they'll bring it over.

Everyone's there to just like, you know, support the one surgeon.

And so the the mythical man month actually predicted that that is kind of the direction that software engineer is going to go.

I don't think that's exactly played out where like you know, it's much more collaborative and like it's not only one person doing the work.

But I've always really liked that analogy and and and that analogy is actually what I strive to kind of like emulate in my own management philosophy, which is.

Software engineering isn't really like surgery where it's not just one person doing work.

But the way in which I like treating the people on my team and the way that I act as a manager is I want to empower them make them feel like they're a surgeon.

And and in so far at like as like making sure that I'm supporting them making sure they have everything that they need to do their work and it feels like they have an army of people kind of supporting them.

And looking around corners and giving them everything that they need when it's really just me as the as the manager and so like the example that I give is is looking around corners and unblocking people, especially from an organizational perspective is extremely extremely useful.

And again going back to the eye conversations even more important nowadays right like if people are just like cranking PR after PR the main thing bottlenecking progress and and you know shipping something tends to be organizational or like process oriented.

And if you as a manager and kind of look around corners and kind of unblock the team if you can you know like if if the surgeon needs scalpel but you know the manager kind of already has a scalpel ready for them that that's the best case scenario.

And it's kind of the way that I approach.

Management and especially engineering management and so that's something that that's really really stuck with me over time and even though you know.

Software engineers aren't exactly surgeons that metaphors always kind of stayed in my mind as up as of.

Forrest my career.

I love that and I feel like I wonder if that's something I can help with is look around corners and predict here this engineer is going to be blocked by this decision we need to figure this out.

That's actually a really good point I haven't tried this yet, but I wonder what happened if I ask Chad Gbt hooked up to a company knowledge, you know like what are the active blockers look through all the notion docs what are that maybe slack messages you know it's probably in slack somewhere.

What are the active blockers on my team and is there something I can do to help.

Now that's very I have not thought about that but you guys had an insight right here.

And it's I think even more interestingly, what do you anticipate will be a blocker for this engineer or this team in the coming months.

Yeah, you asked that you asked the model, he asked the AI to do the second and third order.

And just pay that and just pay what the blockers will be next month too.

We've got a we've got a good idea right here.

Yeah.

This episode is brought to you by data dog now home to Apple the leading experimentation and feature flagging platform product managers at the world's best companies use data dog.

The same platform their engineers rely on every day to connect product insights to product issues like bugs UX friction and business impact.

It starts with product analytics where PMs can watch replays review funnels dive into retention and explore their growth metrics where other tools stop data dog goes even further.

It helps you actually diagnose the impact of funnel drop offs and bugs and UX friction.

Once you know where to focus experiments prove what works.

Now this first hand when I was at Airbnb where our experimentation platform was critical for analyzing what worked and where things went wrong.

And the same team that built experimentation at Airbnb built up.

Data dog then lets you go beyond the numbers with session replay.

Watch exactly how users interact with heat maps and scroll maps to truly understand their behavior.

And all of this is powered by feature flags that are tied to real time data.

So that you can roll out safely, target precisely and learn continuously.

Data dog is more than engineering metrics.

It's where great product teams learn faster, big smarter and ship with confidence.

Request a demo at data dog HQ dot com slash Lenny.

That's data dog HQ dot com slash Lenny.

Okay.

I'm going to shift to talking about the API in the platform that you all build.

So you work with a lot of companies implementing your API, your platform building on your on your tools.

You told me that you find that a lot of companies actually have negative ROI on their AI deployments.

Which I think is what a lot of people you read about and feel and think.

And it's interesting actually seeing that what what's going on there, what are they doing wrong, what is what's happening in the world of AI and deployments in our way.

Yeah, so to be clear, I don't like explicitly see quantitative numbers around this.

You know, it's actually really hard to measure these things.

But especially from observing some companies kind of trying to do AI, I would not be surprised if a lot of AI deployments are actually negative ROI.

I mean, part of this, who is I think there's also general sentiment from folks around the country, like basically outside of tech that AI is being forced onto them.

And I think part of this is is is probably a symptom of some negative ROI AI deployments.

A couple of things I've observed around this.

So one one thing is and I think I come back to this again and again.

Like I think we in Silicon Valley just forget that we live in a bubble like we are so like Twitter is a bubble is our ex is a bubble Silicon Valley is a bubble software engineering is a bubble most people in the world most people in the US are not software engineers are not very AI codes are not following every single model release.

And so and so we're just like highly out of the loop on how to use this technology and so you know, like we we always talk about all these like best practices for codex all these codex built people within open AI.

Sure, everyone on x2 posts are like crazy power users of these AI tools, you know, they lean into skills, they lean into agents dot MD MCPs.

Yes, yeah, all all that and when I talked to some of these companies and I talked to the the actual employees using these it's like the most basic thing that they're trying to do and they like have very little understanding of exactly how the technology works and so that that's that's kind of like one big observation for me which is like.

They're asking very simple questions of these things they're really not not pushing it just yet and so that kind of goes back to that's kind of ties into to what I what I think.

More companies do or like what should do or what what a more ideal AI deployment setup looks like.

And and this is kind of how we've run things within open AI too.

The companies where I think it's it started to work really well have a combination of both top down by and so it's like the sea switch like you know we're we want to become an AI AI first company.

So there's buying they buy the tools they have you know exact support, but it also has bottoms up adoption and buy it and so what I mean by that is it has like.

That's like actual employees doing the work who are really excited about the technology and are willing to learn evangelize build best practices and kind of like knowledge share within the organization.

We've we've seen this a lot internally so like obviously opening I has always wanted to be a very AI centric company.

But where when it really started taking office when was with the introduction of codex and these tools were like people didn't like actual employees themselves could start applying it to the work.

And I think you really need this because at the end of the day everyone's work is like very different it's like very unique software engineering is different than finances different than operations is different than go to market and sales.

And so there's like a lot of these like last mile intricacies of work that needs to really be done in a bottoms up fashion.

And so my sense is a lot of these these air deployments don't have like don't have bottoms up adoption like it was like an exact mandate and it's extremely top down and is very divorced from what the actual work looks like.

And as an end result you end up with a giant workforce that doesn't really understand the technology is like I know I'm supposed to use this and maybe it's like on my performance review too, but I'm not sure what to do and they look around no one else is doing it there's no one else to learn from.

And so my my you know my recommendation for companies kind of pushing this is designed or maybe even staff of full time team internally that is this kind of tiger team internally that can.

Explore the full extent of the capabilities apply to specific workflows do the knowledge sharing.

Create excitement within folks who might want to use a technology because in the absence of that it's very difficult to is actually very difficult to pick up.

And who would you put on this tiger team is it like engineer led you find in your experience is a cross functional sort of team.

Yeah, it's it's interesting.

Also a lot of companies don't have software engineers and so the pattern of scene is that tends to be these like.

Software engineering adjacent like basically technical people but are not software engineers I think there's a lot of time to get most excited around this it's like you know maybe the it's like maybe the like you know support team operations lead.

Who doesn't code but loves using these tools and you know is like an excel wizard or something.

And so it's like technical adjacent or like coding adjacent and like you know pretty technical those are the times it like those are the kinds of people have seen in these companies who just like really light up and get excited around this.

And you can usually build a team a team around that but yeah it's like oftentimes not software engineers software engineers I think we'll understand this but not every company has a software engineers is actually kind of rarity they're they're they're hard to find their expensive.

And so it's these other other types of folks.

What I'm here is the anti pattern is top down this is very the CEO found an exact team just like we are going to go AI first we're going to lead into AI everyone's going to be judged on their performance using AI tools.

How much your productivities increasing things to AI and without without with that being just top down and not creating a team that is bottom up spreading the the gospel you find it doesn't work.

Yeah exactly and the advice is find the people that are most excited and instead of kind of having them spread out through the organization you're what you find works is.

Create a little AI kind of evangelist team that yeah finds ways to use it and kind of spreads it across the work.

Yeah, I mean another kind of like hearing you you play back to me another way to think about it kind of time back to my own.

Imagine a philosophy is find the high performers in AI adoption and empower them you know let them build half of ons let them.

You know hold seminars do knowledge sharing kind of create the seeds of excitement internal okay amazing.

There's a couple of hot takes I want to hear from you something that I've seen you talk about and share one is.

You've shared that talking to customers and listening to customers not always the right strategy in AI and it might often lead you astray.

I don't know if it's that hot of the take I think the main thing here is so obviously you should talk to your customers like it's it's like useful to talk to customers.

I just think that AI field especially what I've seen over the last kind of like three years.

Working on the API and and seeing kind of all that of all is the field and the models themselves are just changing so so quickly.

They tend to like disrupt themselves especially around the like tooling and the scaffolding space so.

There there's just quote that I read actually earlier this week from a it's from an ex article as kind of Nicholas who's the founder of a star called thin tool.

Where I think he was sharing a lot of the best practices that he has learned through building AI agents for financial services I think it is at a certain tool.

I may add this phrase that I thought was really good, which is the models will eat your scaffolding for breakfast like if you look if you're a wine back to 2022 right when Chad GBT launched.

These models are pretty raw and there was like all this product scaffolding and things especially in the developer space to basically try and steer the model and build a scaffolding around it to get it to do what you want.

Like agent frameworks there's like like vector stores I think was like really popular back then and just like a whole smattering of tools here and as you've kind of seen the field play out that the models have just changed so much.

That and gotten so much better that they end up yeah literally eating some of some of the scaffolding and I think this is even true today so I think the the article from Nicholas actually you know that the current scaffolding which is a fashionable skills files based context management.

I could see a world where at some point you know that's no longer useful where the model can actually you know manage all that themselves or like you know or or there might be you know it's hard to predict like might move on to some new paradigm where you know or any of this file based skills skills type thing.

You have literally seen this play out or like the agent framers I think are a little less useful now.

There's a period time like 2023 where we thought vector stores and is is going to be like the main way for you to you know bring organizational context into the models and you need to you know.

Vectorize and in bed every bit of your corpses and then you do all this work to like figure out the vector search to like optimize that to fill out the right information all right time.

All of that is scaffolding because the model you know was not good enough and turns out you know in this case it turns out as the models get better.

A better approach is actually to take out a lot of that logic and trust the model and give it a set of tools for search.

It doesn't need to be a vector store you could actually just hook it up to any type of search.

You could literally be files on a file system like skills in agents MD to kind of steer it as well obviously they're still a place for vector stores.

I know a lot of companies still using it but the the entire scaffolding around that and building an entire ecosystem around that and assuming that the only scaffolding that you need has has really changed.

So tying this back to the like you know it's you know you don't always have to listen to your customers because the field is changing so much at any point time you know a lot of people are kind of in this local local maximum.

And if you just blindly listen to your customers they'll be like yeah I want a better vector store like I want a better.

I want a better you know agent framework for this and if you had just kind of only chased down that path it actually would have led you to you know build something that again is the local maximum.

Whereas as the models get better we've had to reinvent and kind of rethink the right right abstractions and the right tools and frameworks to build around these models.

And the cool slash exciting slash kind of crazy annoying part is it's a moving target and so yeah like the current current smattering of tools and frameworks right now will likely need to evolve and change pretty significantly over time as the models gets smarter and better.

But that is just nature of building the space I think that's what makes it exciting but it also means when you talk to customers you kind of need to balance the exact feedback that they want.

With where you think the models are going and where you think things will trend over the next one of the years.

It's interesting how this is the bitter lesson is you know this big lesson that.

Yeah and ML folks learn which is just like don't the less you over complicate the less logic you add to to machine learning to I the.

More it'll be able to scale and grow and just like take it all the way and let it just just compute basically just give it more power to to get.

Yeah, there's literally a version of the bitter lesson applied to like building with AI where you know we were trying to architect all this stuff around and turns out the models are just kind of you know you did all the way.

And and and and honestly like open AI API team has like been guilty of this where we kind of like took some you know left and right turns when we shouldn't of.

But yeah the model still end up models get better and we're all learning the bitter lesson day and a day out.

So what would be the key takeaway for folks building on say the API are just building agents and you know having to build a little bit of surround for now is it just yeah what would be the advice.

Like general guys and I've been giving this to people for a while and I think still true today is make sure you're building for where the models are going and not where they are today.

You know the the it's it's clearly moving target and I think a lot of the companies that I've seen start up that I've seen really really do well is they build a product for an ideal like type of capability.

That is like maybe 80% of the way there today and it like they end up you know having a product like kind of works but it's like just almost there.

But then as the models get better you know suddenly in my click and then their product now is incredible because it works you know like.

Like maybe like oh oh three at some point to send the risk with 5.1 5.2 suddenly it unlocks it but they're building these products with the like the model capability improvements in mind.

And with that you end up creating an experience experience that's way better than if you had assumed that it's static in the first place.

And so that would be my general advice which is you know build for where where the models are going and not not where they are today you end up building a better products.

You may need to you know like wait a little bit but like you know the model is getting so much better so quickly you often don't need to wait that long.

So it's as full of that thread.

Where are like in the next 6 to 12 months.

Where is the API heading where is the platform heading where the models heading as much as you can share and now there's a lot of secrets here that maybe you're more succeeded better you think that people should start to.

Prepare for however much you can share.

I mean so the obvious one is how long of a task these models can do coherently so there's like the the meter benchmark that that I think track software engineering tasks and how long.

You know like how long of a task can these models do 50% of the time 80% of the time.

Think we're at something like multi hour tasks being able to be done by software engineering cast being able to be done by.

These frontier models 50% of the time and then I think 80% is something like just under an hour.

But the the sobering thing about that that chart is they plot all the previous models on this chart as well so you can really see the trend of this.

That's something that I'm really excited about which is you know I actually think products today.

Really optimized for tasks that the model can do for like minutes at a time like even codex and like the coding tools I'd say like.

You know it's it's in the client you're kind of like seeing it be interactive it's really you know quite optimized well for like maybe almost 10 minute.

Type paths I have seen people push codex a little bit until like multi hour long tasks but again I I think that that's more of the exception.

But if you follow this trend like I think like in the next 12 or 18 months we could see models I could do multi hour long tasks very very coherently.

At some point in my reach like you know six hours a day long task where you kind of like dispatch it and have it do you know do things on.

For a while the types of products you build around that will look very different you want to give the model feedback you obviously don't want to completely run wild for a day maybe you do.

But you probably don't and and then the the universe of things you can have the model do really expand so that's something that I'm really.

Really excited about seeing another thing over the next 12.

I think it's really cool is permits in arm in the multimodal models so and and actually by multimodality I'm mostly thinking about audio.

Here where the models are pretty good at audio I think they're going to get a lot better at audio over the next six to 12 months especially the likes you know the native multimodal.

The speech speech ones I think there's also interesting work being done around new types of models and architectures on the multimodal audio side as well.

But audio especially in the enterprise and in business setting I think is a hugely underrated domain still like everyone talks about coding it's all text.

But we're talking in audio a lot of the world businesses done the audio a lot of services and operations are done via talking in audio and so I think that that area is going to look very exciting the next 12.

18 months and I think there will be even more on mock for what we can do with with audio models there is.

Amazing so quick summary expect agents and AI tools to run longer to that trajectory to continue to increase and then audio and speech becoming a bigger deal more first party and native and better and core to the experience.

Yeah extremely cool okay I want to go back to one of your hot takes another hot take that I've seen you discuss your big you're very bullish on business process automation as an opportunity in the world of AI talk about that.

Yeah this goes back to the thing that I said previously which is we we we live in a bubble in Silicon Valley and a lot of the work that we do.

That we're used to software engineering you know product management building products is very differently shaped than the work that goes on that runs our entire economy.

And I see this in and out when I talk to customers if you talk to any like company that's not based in it's not a tech company there's a lot of business processes.

And so what I mean by this is is you know I generally delineated as you know there's like like software engineering is kind of like open ended knowledge work.

Right it's like and this is why I think tools like codex to be quite quite good because it's exploring and you're giving up these like open ended things but software engineering is fundamentally like pretty open ended and it's not very repeatable right so like you build a feature you're not trying to build the exact same feature over and over again.

And a lot of like tech jobs are in the space I think like data science is kind of in the space as well even some of the like strategic finance stuff.

But as you move further and further away from software engineering and like what is current check a lot of jobs are just business processes they're like repeatable things repeatable operations.

That's you know some manager at a company has kind of like iterated on there's usually a standard operating procedure that people want to do.

And you don't want to deviate from it that much you know this like in software engineering the ingenuity is is in deviating but a lot of a lot of the the the work thing done in the world is actually just running through these procedures and operations like if I you know if I call.

A support line they're running through one of these if I call my utility company there's a bunch of processes and things that they can and cannot do for me.

And so I'm I'm just extremely bullish on this general category of like and and and I think it's underrated because it's so different from what we think about it in Silicon Valley people tend to not think about it.

But how can we apply AI and and some of the tools and frameworks that we have towards this business process automation.

Towards automated automating and making easier repeatable business processes with high determinism.

That is fully integrated with business data and business decisions and and different systems within an enterprise and how can actually make that process better.

Because I actually think there's a lot of opportunity and a lot of work to be done in that area and we just we just don't talk about it because it's it's a little bit less in our real house.

So you take here just to make sure and fully understand it is you think there is a much bigger opportunity outside of engineering for AI to impact.

Productivity companies and also jobs of these folks that are doing these kind of repetitive easily automated tasks impact jobs and also just impact how work is done like so much of work is done in this way like you think about.

Basically I talked to customers all the time big enterprises like how how will AI transform my company like how will it run in in a world with AI in like 20 years and and you know software engineering is part of the story.

But there's so much more on the business process side and I actually think it might look even more different on the business process side and and the work there's is pretty substantial.

It's interesting I don't know like from an absolute percentage or absolutely based I don't know if it's bigger or smaller in software engineering like software pretty huge and pretty expensive as well, but it is pretty massive and it's definitely bigger than you know.

It's it's bigger than you would think it is based off of how how people talk about it or don't talk about it on X or Twitter okay in going in a slightly different direction.

Having built a platform building the API people building on API and the biggest question on people's minds as always just.

How do I not have open AI squash my idea and build their own thing and then you know just try this this market I created.

What's the general policy with the general philosophy of how startup should think about where open AI is unlikely to go my general answer here is is.

The market is so big and so massive like I actually think you know startups should just not overly think about where open AI or these labs are going.

I've talked to a lot of startups you know that have you know not worked out starts that are doing really well.

Every sort of that of scene that is kind of fizzled out is not because open AI or you know big lab or Google or something has has come to watch them it's because they built something and it like really didn't resonate with with the customers.

Where's the ones that take off like even in very competitive spaces like coding like curves are huge at this point and just because they build something that people really love and so my general advice is like don't you know.

Really stressed about this just build something that people like and you will you will have a space in this.

I can't overstate how big of an opportunity there is right now like.

The the opportunity space in building with the AI so big like a good example this is is like the space is so big that the overturned window of what is acceptable and not acceptable for species to do has completely changed here.

VCs are like investing in like competitive companies left and right is just like the space is so big because the opportunity is is unlike anything that we've seen before.

And while you know that that affects our VCs operate from a start of perspective it's like the most empowering thing in the world because the like even if you just build something that that some people really really love you will you will end up with the massive massive valuable business.

And so that's why I tell me like don't don't know what you think about it.

The other thing like I also think is important to remember at least from an opening eye perspective one thing that that we've always held very near deer which both salmon Greg help you know reinforce from the top as well.

Is we actually view ourselves fundamentally as a like ecosystem platform company the API was our first product.

We think it's really important for us to foster this ecosystem and continue to you know support it and not squash it and so if you kind of look at the decisions we make this is all we've we've threw it.

Every single model we've released in one of our products gets released in the API like even you know we release these codex models now that are a little bit more optimized for the codex harness.

But they always find their way into the API and like all of our you know customers end up using those we don't hold back on any of that.

We think it's really important to keep our platform neutral and so you know we don't block competitors we allow people to have access to our models.

We also want you know like we've recently been testing more like the sign in with chat GPT you know product as well and so we we want to foster this ecosystem and things really important that we do so.

The general like thinking about this is like you know a rising tide like lifts all boats and you know we might be a aircraft carrier like pretty big at this point.

We're going to raise the tide because everyone kind of benefits and I think we'll benefit as well like our API itself it's grown pretty significantly because we we act in this way and so.

I'd really encourage people not to view up hi is this kind of like you know being that all just a shelf people out of the way.

But it said focus on on building something valuable and we you know remain committed to providing an open ecosystem.

Is that important to open AI just this focus on building a platform creating a way for people to build businesses just like is that just that's been the vision for the beginning we want this to be a platform.

It's been the vision from the beginning it comes goes back to our charter actually like our mission.

So the open as mission has always been to want to build a GI so we're going to where I was eating that but then second things is like spread the benefits of it to all of humanity and there's kind of like a lot of you know.

The main part there is all humanity like and obviously chat you but do you try to do this you know we're trying to reach however many you know the whole world.

But very early on and this is why we we launched API back in my things like 2020 or something like really early we don't think we as a company will be able to reach all of humanity right like there's I don't know every every corner of the world like like pretty pretty pretty deep and so.

We actually feel like in order for us to fulfill our mission we need to have some platform.

I think here where we can empower other people to build you know the customer support bought for podcasters and newsletter hosts because we're not going to be able to do it ourselves.

And so we've largely seen this play out with the API this is why we we you know we we we we talked to some of our customers and and really you know loves seeing that of our Steve things built on.

But yeah it's been there's to say one because it's it's kind of we view it as an expression of our mission.

Even mentioned the apps for the guys are launching the chat GPT app store yeah is that into your umbrella by the way is that a different work and team.

It's a it's a different team so it's under catchy PT we obviously collaborate very closely with them and you know they built like an apps SDK which is built in close collaboration with 13.

But that is more within the chat GPT umbrella but that is also another like that's another example of this right it's like.

chat GPT is like we we we we kind of like have these 800 million weekly active users who are just coming over and over again like.

It's a great asset to have as a business but like man would it be better if we could somehow allow you know other companies to come in and and and take advantage of this as well and and build for the.

And then ultimately we think it will help us expand that that that group as well right and so it's all it all kind of comes back to the mission and we find that being a platform being open tends to help here.

Just that number 800 million I think it's MMAs just like.

Yeah, it's a crazy billion people using weekly this like it's absurd.

How many know how these numbers were just used to now but that's in insane unprecedented.

Yeah, it's mind bottling for me to think about from a scale perspective.

Honestly, and the way thing about is like 10% of the world and growing by the way, like it's just it's shooting up.

Come to chat GPT and use it every day or sorry, every week and this point I just want to double down on this point you're making.

Open AI's mission was to make AI available to all of humanity and I think some people just that they're like, oh, you know, cost money and it's like.

Like the fact that it it's there's a free version of chat GPT that anybody can use that is not so different from the most powerful AI model that exists in the world for free.

It's not related that anyone can use like if you have if you're a billionaire there's only so much more you can get out of AI that what someone, you know, and a village in Africa can get and and now that's always been really important to open AI.

Yeah, yeah, I mean, look, that's why I think we've leaned into the health work, we've leaned into like like, like education is going to be a very interesting here.

In saying kind of trend here is is the free model has gone so smart over time, like the free model back in 2022 was, you know, like, well, it's good at the time, but it's like nothing compared to what you get today because you do be five today.

And so the like, you know, raising the floor across the world is kind of, you know, something that we're really trying to do and then we viewed as part of our mission.

The other flip side of this by the way is like, you know, kind of talking about like the billionaires or whatever.

I know people have saying like you're using the same iPhone that like, you know, Steve, or sorry, like Mark Zuckerberg's probably using her like the billionaires are using.

Like for like $20 a month, you're basically using, you know, like using the same AI that, you know, the billionaires are using for like $200 a month.

You get the same pro model that, you know, all the billionaires are using, but they're probably not using pro for everything, they're probably just using the plus tier ones for their day and day out.

And so yeah, this kind of like democratization and just like spreading of this, this benefit like across all of the world is send us really meaningful to us and something that.

drives a lot of of what we do.

One last question, just for folks that are thinking about building on the API are just like, oh, I could do cool stuff with open ass models and APIs.

What, what does your API and black form allow people to do like I know you can build agents on top of the platform just talk about what you allow.

So fundamentally, the API offers a bunch of developer endpoints and and and these developer embers basically let you sample from our models.

The most popular one that we have right now is one called responses API.

And so this is an endpoint and it's optimized for building long running agents so agents that will work for a while.

So what you can basically using, you can, you know, at a very, you know, low level, you're basically just giving the model text the model will work for a while.

You can kind of, you know, pull it to see see what it will do and then you'll get the model response back at at some point.

That's like the lowest level primitive that we have for people and that's actually what a lot of people use that's the most popular way of building on top of API with that.

It is like super unipeninated and you can do basically whatever you want like the lowest level thing.

We've also started building more and more kind of like layers of abstraction on top to help people build some of these.

And so next layer up we have this thing called the agents SDK, which has also gone extremely popular.

This allows you to use, you know, the response API or some other API in points that we have to build what you might more traditionally think of as an agent like a, you know, and I kind of working in an infinite loop.

It might have sub agents that it delegates to it starts building all this framework all the scaffolding actually, you know, we'll see where this all goes.

But it makes it a lot easier for you to build these these these kind of agents giving it guard rails allowing you to like farm out sub tasks other agents and kind of like orchestrate a swarm of agents.

The agents SDK kind of allows you to do that.

And then above that, we've now started building tools to help also with kind of like the meta level of deploying an agent.

So we have this product called agent kit.

And widgets, which are basically a bunch of UI components that you can use to very easily build a very beautiful UI on top of either our API or agents SDK.

Because, you know, a lot of times these agents kind of look very similar from a UI perspective.

And so there's Asian kit, we also have a smattering of like evalves products like evalves API where if you want to test and like, you know, see if your models or your your agent or your workflows working.

You can test it in a very quantitative way using our evalves product.

And so yeah, that I viewed as like these various layers, they're all kind of helping you build on what you want with our AI with our models.

And with increasing levels of abstraction and and and and and you know, how opinionated this.

And so you can stuff you can do that you can use the whole stack and and it very quickly allows you to build an agent.

Or you can go down down the stack as low as you want to the base of their responses API and and build whatever you want because of how low local this.

Sure, when is there anything else that you want to share anything else you want to leave listeners with anything we haven't touched on that you think might be helpful before we get to a very exciting lightning round.

The only thing I'd leave folks with is yeah, I think.

I think the next like two to three years we're going to be some of the most fun in tech and in the startup world that that will have in a very long time.

And I would just encourage people to not not take it for granted like I entered the work force of 2014.

It was great for like a couple of years.

I felt like there was like a period of like five to six years where it wasn't very exciting and tech.

And then in the last three years it's been the most insanely exciting energizing period of my career and I think that I suited three years going to be continuation of that and so.

When Kurt will not take it for granted, I'm trying to not take it for granted at some point, you know, this way is going to play out and it's going to be a lot more, you know, incremental.

But in the meantime, we're going to get to explore a lot of really cool things and then a lot of new things and change the world and change how we work and so that's the main thing I'd leave folks with.

I love this message.

I want to spend a little more time on it.

When you say don't miss it, is it what do you recommend people do is it just build lean in learn.

Join a company building really interesting things like what's what's your advice to folks that are like, okay, I don't want to miss the boat.

Yeah, I would just say engage with it.

So it's basically like what you said lean in building tools on top of this is part of the, you know, it's part of the story.

Just using the tools like you don't, you know, you don't need to be a software engineer to lean into this.

I think a lot of jobs are going to going to change here.

So just using the tools understanding the limitations of what a can and cannot do so that you can kind of watch the trend of what you can start to do as the models improve.

And yeah, and so it's basically like getting used and getting used to this technology and getting familiar with it instead of kind of like laying back and letting it, letting it pass you.

On the flip side of that, there's a lot of, I think, stress and just anxiety around like there's so much happening.

How do I keep up?

I got to learn a lot about this week.

Oh, God.

Yeah, what is there something you learned about it just not like you're at the center of this.

How do you not get overly stressed and worried about missing things that are going on and just key stand top news with what are some things you've done learned.

Yeah, so I think I'm personally a bad example of this because I'm basically chronically online on X and our company Slack.

So I actually try and absorb.

I end up absorbing a lot of it.

What I will say, though, just like from observing other folks were less, you know, addicted to the stuff like I am.

Yeah, a lot of it is noise like you don't need to, you don't need to have like 110% of this kind of past your mind like going to your mind.

Honestly, just leaning into like one or two different tools starting small is already like, you know, more than you need here.

I think just the combination of like the frenetic pace of the industry acts as a product just creates like this insane kind of like.

Yeah, this insane like pace of news, which is honestly very overwhelming.

The main thing is like, you don't need to be, you don't need to know all of that to really engage with what's happening right now.

And even something in simple is just like install the code X client play around with it.

Install, try to be keen connected to a couple of your, you know, internal data sources, notion, Slack, GitHub, and see what it can and cannot do.

All of that I think is a part of it.

Amazing, Sherwin, with that, we reached our very exciting lightning round.

I've got five questions for you already.

Yes, yeah, absolutely.

First question, what are two or three books that you find yourself recommending most other people?

Oh, all the time about one nonfiction one and one fiction book.

The fiction book was I just finished reading it.

I, it was really, I really recommend it.

It's, there is no anti mimetics division by qntm.

I think it's like an online author, but I saw it being shared on X.

This, this, it's like a science fictiony kind of book.

And it was, I basically devoured it in like two days.

It was, it's super, super, well, and super fascinating.

It's about a government agency that's fighting, you know, things that make you forget it.

And so it's just a very, like smart, like creative book that, and fresh, honestly, in terms of like source material, that, that I really like.

So I, I'd recommend that one.

The book is also unintentionally hilarious.

So I think it's like meant to be like this, like sci-fi, almost like horror.

It's all book, but it was, it was a, it made me laugh a couple times.

So that's the, that's the fiction book.

Nonfiction, so I'm going to cheat, and I'm going to recommend to them.

So in the last year, I've been reading a lot more about China and kind of like the US China relations.

And I think there are two books that came on the last year that have been, you know, really, really eye opening for me.

And, and in that regard, first one is the Dan Wing book breakneck.

That one is really, really good.

I really liked his analogy of like the lawyerly, US is the lawyerly society.

China is the engineering society.

And they're pros and cons to each.

I read it and I was like, hmm, yeah, it does.

Does seem like we're run by the lawyers in the US.

So that's one.

And the other one is the Patrick McGeebook on Apple and China was super super super interesting.

I'm a huge Apple fanboy.

Like if you could see my desk right now, it's all Apple stuff.

But just like, one, it was just super fascinating learning about Apple's relationship to China.

And then, too, it's just like had a lot of inside information about Apple as a company that I found fascinating.

So it was also quite a page turner and also, you know, very, very timely and timely booksball.

The anti-magnetic book sounds amazing.

I'm buying it right now as you're talking.

Yeah.

Yeah.

It's like, I think it's only like a couple hundred pages.

I literally finished into the dream.

It was just like so.

Okay.

Great tip.

And movie or TV show.

You have a really enjoyed.

Yeah.

That one's tough because, you know, with I have two kids and a busy job.

And so I really haven't had much time to watch TV shows.

I will say in the last couple of weeks, I watched a couple episodes.

I'm actually a big anime guy.

And so I watched a couple episodes.

There's a new season of this anime called Jiu-Jitsu Kaisen.

That's out.

So the season three of JJK was was really good.

In general, I'm a huge fan of Japanese anime.

I think they create the most novel and unique plots in the universe.

That western media has shied away from.

And so generally big fan of that.

But yeah, it haven't really watched much.

But so a couple episodes of JJK recently.

Extremely understandable in your role.

Yeah.

Favorite product you recently discovered that you really love?

Yeah.

So I recently had a setup a Wi-Fi and like home networking.

And I went all in on ubiquity routers and cat security cameras.

I'd never heard of it before.

I had to do this.

I always had a very simple setup.

And it's just such a well built product.

I don't have a used it before.

But it's basically like the apple of like home networking.

So beautiful products.

But the thing that actually makes it extremely good is that software is good.

And so they have a really great mobile app to help manage, you know, all of the home networking.

And so basically ubiquity you can use it to buy wireless routers.

You need Ethernet wiring throughout your house to use it.

But I think what makes it really good are security cameras.

So if you have security cameras that are plugged into the ubiquity ecosystem, they have an incredible mobile app.

And Apple TV app and iPad app to kind of see the live feed of your cameras.

And so they're they're they're a little pricey, but not that pricey.

But it's been just an incredible product experience.

All right.

I went euros.

So I made a mistake.

Because euros are pretty good too.

But I know ubiquity fully converted to ubiquity.

Okay.

Okay.

Two more questions.

Do you have a favorite life motto that you find yourself going back to in work or in life?

Yeah.

The one that I always, you know, repeat to myself is never feel sorry for yourself.

There's a lot of things that are going to happen.

You know, at work in life and reminding yourself to never feel sorry.

And that you always have a sense of agency to kind of pull yourself up.

Is something that I've had to tell myself a lot.

And also something that I repeat to a lot of other folks as well.

Last question.

So in your previous life, you worked at Open Door where you let work on.

Basically figuring out how much to pay for houses.

You basically build the model that told the company here so much will pay for the house.

What's like a variable in the price of a house that you didn't expect.

That you didn't expect is really important and impacts the price of a house.

There's a bunch that we're surprising.

I'll maybe list the couple of most interesting ones.

Power lines and like high voltage power lines.

Like our super super actually impact your price quite a lot.

I didn't really fully internalize this until I went to like Dallas and observe like when your house.

It's next to one of these giant like, you know, voltage lines is like buzzing and most people have families.

You don't want your kids kind of near there.

So I think that was one that really really kind of surprised me.

That makes us.

Yeah.

And then the other one which was something that was always something really difficult for us to quantify was floor plans.

And so it is very important, like yes, of course it's really important.

But just like quantifying what a good floor plan is like what a really bad floor plan is like.

We were doing all these things like how wide is the kitchen and like is it a what solid kitchen is it and then like where's the master bedroom and so it was just really really hard to quantify.

But I remember floor plan was a big one because like we'd have a home that like wouldn't sell and then our ops team would go in and be like, yes, the floor plan to shoot.

So like how could you tell us like you go inside and you just feel it.

It feels you know the floor plan to feel it feels off.

So yeah, those are ones that were so rising.

And then the last one that was more impactful than I thought is general like curb appeal and like even like the front door.

And so I think there's a little book on on this where the front door placement tends to be the highest ROI for homes.

But just like the feel of like as you walk up to the home as a buyer.

What year interacting with and the first moments of the house I think was I had underrated hits and parts.

That is extremely interesting.

And I love that you had to figure out how to do all this encode and not.

Yeah, and then floor plans.

I would bunch of stories around like for floor plans.

There's like there's like it's not digitized.

So there's like a handful of people who have like paper floor plans of like all these homes and like Phoenix and Dallas.

Yeah, a lot of fun on stars from local nerdies.

Okay, sure when thank you so much for doing this.

This was incredible.

Working folks by the online and and how can listeners be useful to you.

Yeah, so I'm online on Twitter on X.

I'm just at sure when we will.

And yeah, I mostly just tweet about open AI and the API and some of the price that we're launching.

And then how folks can be interested.

Can be useful to me.

I love hearing about things that people are building.

And so if you're working on a startup if you're hacking on an idea.

You know, would love to reach out to me on X.

I would love to hear about what you're building.

And and learn about how open AI can help support you.

Amazing.

Sure when thank you so much for being here.

Yeah.

Thank you, lady.

Bye, everyone.

Thank you so much for listening.

If you found this valuable, you can subscribe to the show on Apple podcast, Spotify or your favorite podcast app.

Also, please consider giving us a rating or leaving review as that really helps other listeners find the podcast.

You can find all past episodes or learn more about the show at lenniespodcast.com.

See you in the next episode.