GPT-5-Codex-Max: Training Agents with Personality, Tools & Trust

Source material

Transcript

(upbeat music) - Okay, we're here at EIE code, and we have two of our speakers, Bill and Brian.

Welcome.

- Hi.

- In Space.

- Thank you for having us.

- Bill, oh Brian, I know you've been the lister for a little bit.

- Oh yeah.

- What's your take on Lin Space?

How does it, what role does it perform in your functions at OBI?

- Yeah, I mean, first of all, love the name.

- Okay.

- I'm a massive latent space context management person.

- Tell the story behind the name, maybe the chance.

- Yeah.

- So it start, we never had latent space as a name at the start.

It was called L Space.

- Interesting.

- And one of my readers donated the domain name, latent dot space, he's like, you want it?

I'm like, yeah.

- Awesome.

- So latent just like came accidentally.

- See, it was in the ether, but like I didn't have the domain.

- Yeah.

- So I just like called it L Space.

L Space is like a viscoel.

- Nice.

- So it's for me.

- Yeah, no, it's amazing.

- I love it because it's, you're like always on the cutting edge and it goes into a lot of detail about all the things that like I should be keeping up with as part of my job.

And there's so much to keep up with, right?

So there's only so many sources of really good high quality information for what's like happening on a deep level.

- Well, you guys have your own podcast now.

So I'm like a pre-petition.

Yeah, well, I still listen to yours and I still think yours is really good.

(laughing) So you guys, I guess are representing like a start-ups team, Codex, all the things you just launched Codex.

- Yeah.

- The close Max, Codex Jannini yesterday.

- Yeah, we're gonna name names.

- Yeah.

- I do, people do make friends, I think Thibaut was like, yeah, you know, we're good at a lot of things, but not on eBay.

I was like, well, why call it Max?

Was there any like internal discussion?

- Yeah, I mean, it's complicated because it needs to be differentiated from the previous one.

And the idea is like Max can run for a really long time.

We can go 24 hours or more.

I've actually like sort of had it gone for more than that.

And the name is, you know, - Inside Codex on the web.

Is that, how do you, when you say really long time, 24 hours.

- Oh, on my, oh, that's, I think that was on the web inside of Codex, I'm not sure, but I've actually done it on my local computer for quite a bit longer than 24 hours over the course of a couple of days, but closing my laptop and reopening it.

But the name, you know, you could come up with something like pro, the pro is sort of like slower, more thoughtful.

Max is about sort of like speed and maximization, like maximalist.

- So for this motto, you can, it can run for a long time, but it can also actually, for the same types of problems, it can actually get to the right answer faster.

But I can, it's simply better and faster.

- Yeah.

- Yeah.

So I think the part of what you guys are speaking about is the training that goes into something like that, right?

Bigly, people just kind of wave their hands to the RO, but like, what specifically have you learned about, what's a good practice sauce on?

- So I got to, I mean, this sounds weird to say, but I was lucky enough to be really close to the train team while GB5 was training.

And one of the big things that we focused on, Bill was there too, we focused on personality, right?

So it's really important to build trust with developers for like how a model works.

And if a model doesn't act the way that you expect it to do, or if it doesn't work alongside of you as well, you're not gonna really trust it.

You're not gonna get as much out of it.

So for coding, we thought, okay, well, what is the best personality for a coder, for a pair programmer, for somebody you trust, and how do we like eval against that?

How do we come up with behavioral characteristics?

And we came up with things like communication.

It needs to keep you impressed of what's going on while it's working, planning, like come up with a strategy, do some searching around, like figure out context gather, figure out what to do before you just dive in, if it makes sense to, and then check your work, right?

And so these are just best software engineering practices that turn out to be behavioral characteristics, and we can measure the model's performance on those behaviors and grade it that way.

- Yeah, I will say that another key aspects to how we train to the model is you work really, really closely with some of our coding partners.

And a lot of those folks lead on the bleeding edge.

And so they have a lot of understanding of what particularities they need.

And we really focused on sort of those areas and really draw deeply into those.

- Yeah, that's right, especially tools, right?

So like different harnesses have different tools.

Some people have context like semantic search.

Some people have different ways of doing code edits.

And initially, our models are trained the way they were trained to use tools.

And that kind of bakes in a habit.

And so we've been getting the models better at using different types of tools.

- Yeah, it's a lot to follow up on, but I'll go tools first, and then I'll go back on the personality base.

Both the unit ideas wise, I think the communication with the five products just came out was, well, this is the model trait for our codecs, not necessarily your choice, right?

Has that message changed for other startups using the five codecs model?

- Right, no, so codecs is, just to be clear, codecs is the frontier coding model that we have that is optimized for its harness.

The codecs team is very focused on creating a coding agent and they want it to work perfectly inside of the shape of the harness and API that we have.

So they're completely unbounded.

- It's open source, so.

- Yes, that's open source.

- And the model is available in the API.

So that's what they focus on.

- Yeah.

- It is, the conflict is, well, you just said, other startups have other tools, and obviously I know that.

It is possible, like one thing to mention here is, I think we can probably disimpangle a little bit on sort of the codecs apart from the sort of the mainline models.

The codecs models are sort of focused on agents itself, right?

Like the codecs agent itself, the model has been trained with the agents specifically in mind.

It actually turned out to be somewhat, even sometimes easier to integrate because we come into it with a firm opinion on what the sort of best way of using it looked like.

And so some folks that we work with actually really appreciate that we come into it with that opinion.

I will offer the other ones that has more of a general or specific tools that they definitely need.

The mainline model is one that's more general in the sense.

And that's sort of what Brian was referring to when he's talked about GPT-5 tools getting.

- Yeah, so the five line non-codecs is more general across the board.

It can respond to things that are, it's much broader than just coding.

It has coding capabilities that are also mirrored in codecs and they work together to keep that true to it up.

But since it's more general, it does have more steerability to different types of tools.

And when you're implementing tools, the model can get bogged down if it hasn't seen a tool that it's used to.

And it might take more time thinking about how to use it or make more mistakes.

So our recommendation is if you're wanting to go bleeding edge coding focused, pay attention to the codecs line and the codecs SDK and the codecs models, because that's the one that's like really aimed at that.

You'll have to do some work to like look at how we're implementing our tools inside of codecs to maximize its capability without bogging it down.

But like people are having success like bending it in ways that maybe we haven't thought of.

- If you're interviewing on the mind, I always want to pry if.

- Sure.

- Yeah.

- Do you have any examples?

- You say bending in ways you haven't thought of.

- Yeah, so I think, so codecs is trained with terminal tools in mind.

And so what we've thought would be the case is you will essentially only have to like strip out, you have to strip out all of the tools except for the terminal tools.

But we found some like partners of ours like they discovered that what you can do is that you can actually still have a lot of the tools just named in the same way as the terminal tools as well as having the same input and output.

And all of a sudden the tool called performance jumped up by a lot.

- Yeah, and codecs loves Rip-Grip.

So if you make a Rip-Grip tool and tell it to use it, it'll use it.

- So if you call it Grip, it actually does a little bit worse, but if you call it RG, it actually does really well.

- Right.

- Yeah, yes.

This is something that we ourselves only discover.

- This is one of the coolest things about like model training is literally like they develop habits.

It's just like a person does.

Like if you're like working on some podcasting tool, right?

You're really good at editing and then somebody makes you use a different one, it's gonna slow you down.

You're gonna get kind of bogged down and make mistakes.

- Sure, but I don't know if like, yes, that's very humid, but I don't know if I'll call it cool because it's supposed to generalize.

- Right, that's the end goal, yes, of course.

And so that's what we're doing with the five series of models is they're way more general and Codex is focused on maximizing coding and those are the sort of two horizons that we're working on.

- Yeah, awesome.

I wanna go back on personality.

- I know you hate that word sometimes.

- Hate it, it means different things to do.

- Yes.

- And when it comes to people who are like very angry, keen on like model research, model personality is much more like, what I think maybe one of your friends would say, yeah, it's like, it warms your friendliness or your, I agree with this, having people's emotional state forever.

And so it's really jarring when that is also applied to all the agents where like, well, I wanna talk so much about Cilica Valley HBO, which is also called Anton, but it could be paid on the other side.

I think the other thing is also, but what doesn't matter, 'cause you said a lot of things about like, my commentating is that you're gonna use our engagement on that.

Doesn't matter if it's so quantified anyway, right?

Like you're going for 24 hours, you're closing your laptop, you have like the extra high parameter now.

Doesn't matter.

- Exactly.

So here's, we're in this world right now where we're in between a situation where people don't quite, like the models don't quite have the trust of senior engineers or engineers doing like very important work.

And so we found, our customers have found that people really wanna follow along with what it's doing so they can like interject or stop it, or at least understand what it's thinking so they don't waste all the kinds of time like doing a rollout that they have to throw away.

So with the five series, because it's more general and it's just about as good as coding as codecs for a lot of things, we've taught it to be more communicative.

And so it has preembles before tool calls.

It'll say things like, I'm about to go look for this.

Yeah, and you can steer that really well.

I actually really like it.

I have, I've created like a personality.

I tweeted about this.

I created a personality for my coding agent because I really liked my tools to be kind of like fun to work with if I'm in there with them.

And so I have, it's got this like, it gets really excited if we do something together and like, 'cause I wanna wake up in the morning and be like, oh, I'm gonna go work on this project with my buddy, Five One, right?

But some people don't like that.

And also for like you said, long running agentic tasks, that can get in the way.

Like you're burning tokens that don't really matter if it's running in the cloud.

So Five One, you can turn that off.

You can prompt it not to do that.

But the Codex model can't actually do that.

And it relies on the reasoning summarizer to give you that update.

I guess more broadly, why should people know or think about in terms of what it'll be as to if voting models in general, like more broadly than just like you'd be able to experience release, just like what trends are you seeing, what discussions are active.

- Our talk today is focused on talking a little bit about sort of the trend that we're sort of seeing as the abstraction layer really moving, starting to move upwards from the model layer, where's the agent layer.

As I said, we train our models, starting to be a little bit more opinionated, especially with regard to voting model like Codex.

And the models are really good at doing certain things to widen inside of a certain harness, a certain type in search.

And so we're actually packaging that up more closely.

So we're actually shipping this entire agent altogether.

Then you can actually build on top of that agent.

As one of the patterns that we're seeing here is rather than focusing on optimizing with every single model release, you're actually just be able to plug in an agent like Codex into your platform and be able to use an app box.

Yeah, you're seeing Zed use this GitHub VS code, lets you just like package a whole agent to work inside of it.

And that way, like if you're building a coding tool, like Zed, and you don't feel like having a whole team keep up with all every single model release and every single API change and how to update the harness to do different cuts of sandboxing and all that kind of stuff, you can just build one layer above.

And that is actually super powerful because coding is just like one agentic behavior.

It turns out it's a really nice one to start with because you can measure the performance sometimes easier with a lot of other ones.

But it also gives the model the capability, right?

So we started out with like chatbots, like you have any conversation, let's give the chatbot a tool to use.

Okay, so now you have an agent that can like run commands.

Well, let's give the chatbot agent a Codex to use.

So now if it doesn't have a tool, it can make a tool that it needs to solve a problem, right?

So that's like another layer of abstraction and it's not just coding.

You can write software that has an agent that can spin up a Codex instance and write a custom plugin for your software for that customer's API, right?

And so now your software is self-customizable because it has its own team of people inside that can do integrations at launch.

Yeah, solving integration engineering is a DI.

Yeah, one thing, one theme I'm binding at this conference so far, even early, like the first T-ROCs, I think people are starting to really explore sub-agents, agents that use it.

More abstractly, agents that use agents.

And we used to call it multi-agent, I don't know what it is.

Now, I don't know if there's any thoughts on your end about this, where like you could tool call, I guess like a very basic example is what you just said, which is the agents can create another instance of Codex that creates a tool and then drop it, they just use the tool.

Is there a case for skating some agents, CQSari, AEGO?

Yeah, I think so.

I mean, Codex Max was designed for that, right?

So it has its own compaction and context management.

Codex Max manages its own context window.

And so it can run basically forever without you having to worry about it while it's inside of the Codex harness.

And that lets you do a lot of different things.

You can essentially have it hand off its own context to other sub-agents, right?

So letting it sort of like spawn different agents to do more of its work in parallel and all kinds of things like that.

So it's built for that.

I mean, we're just sort of like starting to see the indications of like what that means.

But that's, I think the future and we're really excited about that.

- Yeah, it's really, I think, like as I said, the trend that was sort of serving here, really moving up the attraction layer to the agent layer really allows you to do a lot of cool things like brand new expansion.

So we need a few agents creating new abstractions as things as the long running agent workflow continues.

And right now we're building all the primitives as well as bottles specifically with added mites.

- Yeah, and it's really about moving the threshold up further, right?

Like I was saying before, like I now trust like Codex to do some of my hardest work.

I haven't written a single line of code by hand in months because I know what I can trust it to do.

- You're the first person that said that in the last three or four years.

- Yeah, no, it's real.

I mean, I've actually launched something.

There's an open source project that I did.

There was a Codex upgrade pack for migrating from completions to responses that was totally written by Codex.

And I didn't write a single line of that code.

And now it's out there, it's open source.

- Most of the folks that open AI, well initially when Codex first launched, it was around 50% of folks that opened AI started using it, but now they go, but those folks that open AI.

- That's very true, we use it every day.

The way that we do it is we're really good at evals, right?

Like in order to develop trust and like build a product that can do more than you design it for, which is really what we're talking about here, you're making an agent that can like solve its own problems.

You have to get really good at figuring out how to build those guardrails and evals around, you know, what is it doing?

What is it allowed to do?

And check it in production.

So we have all of this platform tooling now around agent traces and rollout traces and coming up with evals for that and building, you know, graders and all the things you need to sort of like maximize the pipeline so you can let it go.

And then like be like, okay, I don't really like the way it did that.

Great it, have it metapromped itself so that next time it actually does a better best practices.

- What are the biggest, is you use, in terms of where you're using, for example, capabilities that OPI is investigating is Asia Pia Dusty.

Do you say more about that?

Like why is that suddenly a big priority now?

Obviously I think there was, I had this, OPI always did internal evals, but now it's like a team that is more overrefreezing.

I don't need to be me, you know, this random era.

The path to AGI, it really goes through evals.

And well, I'm sorry, that was a little.

- It's so true.

- It's so true.

It was repeated way too many times.

But I think there are a lot of academic evals, right?

There's like sweep ends, there's other like, you name it.

But I think there's a slightly lack of evals off the real world on sort of what people care about the most.

And we want to make sure that whatever we're developing, model wise, as well as product wise are aligned and are actually making the most amount of useful impact on this world.

And applied evals is really in that direction, capturing all of those sorts of real world use cases and things for us to Hill Climb together.

I like to think of it as like we have, I mean, people say it's a PhD in an API, right?

But if you hire a PhD student, they don't know how to do the job.

You have to give them a job description.

Okay, that's a prompts, right?

So now you have your policy, and then you have them do the job, and they're gonna kind of like flail around, right?

So they need mentorship, they need guardrails, they need evals, performance reviews on how to do their job, the best practices.

And so what we're doing is we're trying to put our models out there and see what they're good at, what they're not good at talking to our customers.

They're like, "Oh, we could really use your model for more things if it could do this one thing.

Here's our eval for it."

Or help us build those evals with you so that we can see where we're deficient and go back and train the model to be able to do that job in the way that we wouldn't normally get to see it form.

- Yeah, how do you do multi-turn evals?

'Cause I think that's the really hard thing that, I mean, sometimes you need multi-turn if it doesn't get around the first go, but it could just get around the first go, then it's no longer multi-turn, right?

So then what?

- Do you wanna take, I have some ideas.

- Oh, yeah, you go.

- I mean, I've built a few myself.

I don't, this is sort of like my personal work.

I think this is like an area that people are just now getting into, right?

We have LM as a judge.

You can use LM as a judge to look at an entire-- - The trajectory, yeah.

- And see, okay, over the course of all of this, like how well it did perform, what did it do?

And then you could maybe like walk it back a step to the part where you don't like, and then you could have the model run the next step with the instructions, grade it on that, and then have it improve itself.

Oh, I don't like the way that you, we do this all the time inside of harnesses.

It's like, that was a good answer, but I don't really like how long it took you to get there.

So can you give yourself better instructions for doing that next time?

It'll write something and we'll add it in there, and then suddenly it's better, right?

So there's, that's one way of doing it.

- Yeah, I think multi-turn evals most of the companies or startups that we work with, like these days, the agent runs and then multi-turn away, right?

And then so therefore if you can build an agentic harness that works in a multiple-turn way, you can eval it.

And then there are like also academic benchmarks already does this in some ways, like cow bench, and now we have like tile square bench that does this like particularly well, and we'll definitely certainly take inspirations from that.

- I have this idea, I call it like a job interview eval.

I haven't finished it, but really like, if you're evaluating a coding agent, what do you want it to be able to do?

You want it to be able to take an underspecified, imagine you were interviewing a developer, you give them a problem, hey, like go implement a string reverse or whatever.

And then it's like up to them to like ask for, okay, well, I need more information.

What are the constraints here?

Like what is, and then you judge them on that.

And then they start implementing it, you give them some modifications, you grade them on that.

You can imagine like building, you know, with an LLM, like a rollout that is promptable and the model responds, and then you can kind of grade the whole thing.

- Yeah, one thing I would love, and this is like the feature request part of the podcast, is batch multi-turn eval API.

So batch API is single turn, but you can't really batch multi-turn requests.

Is that already doable?

- Batch multi-turn requests, I don't believe it.

You can't do it yet, but yeah, I think that's like a really valid.

- Because you need evals to be cheap as possible.

Yes, they're not that time sensitive and you want to run it overnight.

When the things are just, yes.

Well, feedback taken.

- Feedback taken, but that's the thing.

Like every day we're trying to break the platform better.

And right now evals is certainly part of it.

- Literally how we make product feature updates as we talk to people like you.

They're like, hey, can you do this?

- I mean, it's super like, yeah, if I'm going to throw thousands of runs at this thing, you know, I should probably spend some time worrying about costs.

- So you know which, what are you trying to do though?

- I mean, Devin and Cascade, you know, and so I have a personal side projects where I want to make Devin for non-coding.

- Oh.

- I love Devin so much.

Like they Slack, my kind of semi-hot take that I'm floating around, because just to see how it feels, is I think Slack is the ultimate user interface.

- Yes.

- For work, right?

I don't want to read email.

I just read Slack all day.

I interact with my email agent through Slack.

So basically I'm building a Devin for email.

- Yeah.

Well, that's the thing is like you can use, you could use Devin to do that, right?

Like a coding agent, like CODEX, a CLI, it used to be back in the old days.

Like I started out in the nineties working at IBM as a system administrator, and I had to write my own custom software and bass scripts and whatever to actually solve real world problems every day.

And so I had this like, you know, toolkit of scripts that I made, right?

That were like organizing file directories or doing like other random things that weren't necessarily writing code.

- Yeah, yeah.

- And so you can get phenomenal use cases to just like sort through your email using like Elm or something, right?

In the terminal or like have it generate like snippets of video clips from YouTube that you can watch later or things like that.

- You know, I never thought about that, but I do that all the time as part of LaneSpace.

- Yeah.

- I should probably invest in that tooling.

I had CODEX go through my really messy directory of all of these experiments that I was running and like completely organized them and like put them into shape and it was so wonderful.

- I used it for something that's more boring, organizing my desktop.

- Yeah.

- You know, we have a lot of files on the desktop and CODEX is really good.

- Yeah, people think they have all any of CODEX, my mg0416.jpg or something.

- Yeah, well, I'll just find all the images and put them in one folder.

I think that even that's something CODEX can do.

- I think that's one of the big things that we're also seeing like coding tools are breaking out of coding and just like everything, their personal automation.

- Exactly, because the way if you can think about before graphic user interfaces and browsers, like how did we interact with the computer?

Would this go through a terminal?

And we did so by writing commands and writing code and stringing them together inside of the terminal.

So what would you think about it is are those coding agents are actually a computer use agent but for the terminal?

- Yes, yeah.

- They're actually incredibly general.

- I would say that coding agents today are still not vision native enough.

Like you have to try to get it to use vision and oftentimes it fails still.

We should use vision a lot for it.

- Yeah, I would say, you know, I was gonna end the episode with asking for your 2026 traditions.

Like we sit down this time next year, what do you wanna see?

You know, what do you hope to see?

- I'll just kick it off with the easy one.

More computer use.

And I think like when you say things like, oh, we'll have a coding agent build its own integration to your application.

A lot of applications don't have APIs, don't have NCPs.

The only thing you have is a UI, right?

Yeah, because they're legacy or because they don't want you to take the data but while the data is yours, you just have to like get a non-provisioned way ticket by the user.

- Yeah, yeah.

And I can continue just by sort of like saying that that's definitely going to be something, I think is going to be something that we'll be capable of in 2026.

But also the other thing that I am sort of really like looking forward to our codecs being able to do more, right?

We're already starting to talk about our codecs or like coding agents can sort of use computers in novel ways.

We're going to be able to sort of see more general and general use cases like that coming along as well.

And more sensible ways for you to build with those sub-agents as well.

- I really want to see the trust level go up even further, right?

Like I open AI, I get to work with some of the most amazing developers I've ever worked with in my life.

They're incredible.

Like some crazy tech leads.

I wish every company, no matter whether it was like a small dev shop in Alaska where I worked for a while or open AI, be able to have on their team, like capabilities that you would only be able to get at like a top tier firm, right?

So like, so all of my teammates at all of these places could turn to a coding model and be like, Hey, how do we do this like crazy awful refactor that we have to do to get, to support this new customer that we have or like, wow, there's so much of a mess here.

Or like, what's the best way to actually implement this new technology and have it be so trusted and so right and so smart that like, you know we can actually perform better than we could normally get access to.

- Yeah.

I didn't ask her to be any, any final call session.

- Oh yeah.

- We're Brian and Bill at open AI and yeah, feel free to find us on our Twitter socials, whatever.

And then let us know how you're building.

- Yeah.

And we love working with startups and anytime you have feedback about, do you really wish the model could do this or their product can do this and you could unlock some massive capability and just let us know.

- Yeah.

Amazing.

- We'll do.

- That's it.

Thank you, guys.

- Nice.

Thank you.

(upbeat music) (upbeat music)