OpenAI researcher on why soft skills are the future of work

Source material

Transcript

Not only are you working at the cutting edge of AI and LLMs, you’re actually building the cutting edge.

What do you think people misunderstand about how models are created?

Today my guest is Karina Nguyen.

Karina is an AI researcher at OpenAI, where she helped build Canvas, Tasks, the o1 chain of thought model, and more.

Prior to OpenAI, she was at Anthropic, where she led work on post-training and evaluation for the Claude 3 models, built a document upload feature with 100,000 context windows, and so much more.

She was also an engineer at New York Times, was a designer at Dropbox and Square.

It’s very rare to get a glimpse into how someone working on the bleeding edge of AI and LLMs operates, and how they think about where things are heading.

In our conversation, we talk about how teams at OpenAI operate and build product, what skills she thinks you should be building as AI gets smarter, how models are created, why synthetic data will allow models to keep getting smarter, and why she moved from engineering to research after realizing how good LLMs are going to be at coding.

If you enjoy this podcast, don’t forget to subscribe and follow it in your favorite podcasting app or YouTube.

It’s the best way to avoid missing future episodes, and it helps the podcast tremendously.

With that, I bring you Karina Nguyen.

(end of video) This episode is brought to you by Vanta, and I am very excited to have Kristina Casiopo, CEO and co-founder of Vanta, joining me for this very short conversation.

Vanta is a long time sponsor of the show, but for some of our newer listeners, what does Vanta do and who is it for?

Sure.

So we started Vanta in 2018, focused on founders, helping them start to build out their security programs and get credit for all of that hard security work with compliance certifications like SOC 2 or ISO 2701.

Today, we currently help over 9,000 companies, including some startup household names like Atlassian, RAMP and Langchain, start and scale their security programs and ultimately build trust by automating compliance, centralizing GRC and accelerating security reviews.

That is awesome.

I know from experience that these things take a lot of time and a lot of resources and nobody wants to spend time doing this.

That is very much our experience, but before the company in some extent during it.

But the idea is with automation, with AI, with software, we are helping customers build trust with prospects and customers in an efficient way.

In our joke, we started this compliance company so you don't have to.

We appreciate you for doing that.

And you have a special discount for listeners.

They can get $1,000 off Vanta at vanta.com/leni for $1,000 off Vanta.

Thanks for that, Christina.

Thank you.

Karina, thank you so much for being here.

Welcome to the podcast.

Thank you so much, Lenny, for inviting me.

I'm very excited to have you here because not only are you working at the cutting edge of AI and LLMs, you're actually building the cutting edge of AI and LLMs.

You recently launched this feature, which is basically the first agent feature of OpenAI.

I also just did this survey.

I don't know if you know about this.

I did a survey of my readers and asked them what tools to use every day in your work and most used.

And chatgbt was number one above Gmail, above Slack, above anything else.

90% of people said that you use chatgbt regularly.

It's absurd.

It wasn't around two years ago.

Also, we're recording this the week that OpenAI announced Stargate, which is this half trillion dollar investment in AI infrastructure.

So there's just a lot happening constantly in AI.

And you have a really unique glimpse into how things are working, where things are going, how work gets done.

So I have a lot of questions for you.

I want to talk about how you operate and how you work at OpenAI, where you think things are going, what skills are going to matter more and less in the future, and also just where things are going broadly.

So how does that sound?

Sounds great.

Thank you so much.

Yeah, I was extremely lucky to join early days on topic and kind of learned a lot of things there.

And I joined OpenAI around like eight months ago.

So, yeah, I'm excited to that's more interesting.

Okay, I'm gonna definitely ask you about the differences between those.

But I want to start more technical and just dive right in.

I want to talk about model training.

People always hear about models being trained, these big models, how much data takes, how long it takes, how much money toss it takes, how we're running out of data, which I want to talk about.

Let me just ask you this question.

What do you think people most misunderstand about how models are created?

Model training is more an art than a science.

And in a lot of ways, like we as like model trainers, think a lot about like data quality is like it's one of the most important things in marketing is like, how do you ensure the highest quality data for certain like interaction, model behavior that you want to create.

But the way you debug models is actually very similar the way you debug software.

And so one of the things that I've learned early days at Antapriek was like, we've discovered especially with like, cloud training, when you taught the model some of the self knowledge of like, hey, like, you actually don't have a physical body to operate like in the physical world.

But then at the same time, we had data that kind of taught the model some of the function calls, which is like, this is how you set the alarm.

And so the model would get like extremely confused about like, whether it can set an alarm in the fit, but it doesn't have a body in the physical world.

So it's like, the model gets confused.

And sometimes it's like over refused.

So sometimes it's like, I don't know, like, sorry, I cannot help you.

And so there is always like a balanced trade off between how do you make the model to be more helpful for users, but also not being harmful in other scenarios.

So it's always about like, how do you make the model like more robust and like, operate across like, variety of diverse scenarios.

That is so funny.

I never thought about that.

Most of the data that it's trained on is kind of like assuming it's like a human describing the world and how they operate.

And there's, they assume there's a body and you could do things in the model told you don't have a body.

Yeah.

Okay.

I want to talk a little bit about data.

While we're on this topic, I know you have strong opinions here, there's kind of this meme that models are going to stop getting smarter because they're running out of data.

They're trained in a large part on the internet.

And there's only one internet and they've already been trained on it.

What more can you show them about the world?

And there's this trend of synthetic data, this term synthetic data.

What is synthetic data?

Why do you think this important?

Do you think it's gonna work?

I think there are two questions here.

We can unpack one at a time, but people say you're hitting the data wall.

I think people think more in the terms of like pre-trained large models that are trained on the entire internet to predict the next token.

But what actually the model is learning during that process is actually how do you compress the compression algorithm here?

The model learns to compress a lot of knowledge.

And it learns how to model the world.

So the next prediction of the word, teach me how to drive basically.

And you only have a few words that will match that, a car.

So the model actually learns about the world in itself.

So it's modeling human behavior.

Sometimes it's modeling.

And when you talk to pre-trained models, which are very, very large, they're actually extremely diverse and extremely creative because you can talk to almost any Reddit user through pre-trained model.

But I think what's happening right now with new paradigm of like the one series is that the scaling in post-chaining itself is not hitting the wall.

And that's because basically we went from raw data sets from pre-trained models to infinite amount of tasks that you can teach the model in the post-chaining world via reinforcement learning.

So any task, for example, like how to search the web, how to use the computer, how to write, wow, like all sorts of tasks that you're like trying to teach the model, all the different skills.

And that's why we're saying like there's no data wall or whatever because there will be infinite amount of tasks.

And that's how the model becomes extremely super intelligent.

And we are actually getting saturated in all benchmarks.

So I think the bottleneck is actually in evaluations.

That you don't have all the frontier like evas, like, I don't know, GPQA, which is like a Google proof question answering like PhD level.

And the batch of work is like getting to like, I don't know, more than like 60, 70%, which is what HD gets.

So it's like they're literally hitting the wall and like evolves.

I want to follow both those threads.

So the first is on this idea of synthetic data is a simple way to understand it, that the models are generating the data that future models are trained on.

And you ask it to generate all these ways of doing stuff, all these tasks, as you described, and then the newer models trained on this data that the previous model generated.

Some tasks are synthetically curated.

So this is like an active like research area is like, how do you can you synthetically construct like new tasks model to like learn.

Sometimes, you know, like when you develop products, you get a lot of like data from the product and like user feedback.

And you can use that data to like this like post training world.

Sometimes you still want to like use like human human data, because actually, some of the tasks can be like really, really hard to teach.

Like like, like experts like only know like certain knowledge about like some chemicals, or like biological knowledge.

So like you actually need to tap into the expert knowledge a lot.

So yeah, I think, to me, like synthetic data training is more for like product is like a rapid model iteration for similar outcomes.

And you can dive more into it.

But the way we meet canvas and tasks and like new like product features for HTTP was mostly done by synthetic training.

Let's actually get into that.

That's really interesting.

I want to talk about you vowels, but let's follow that thread.

So talk about how this helped you create canvas.

So when I first came to open AI, I really had this idea of like, okay, like it would be really cool for chat ability to actually like change the visual interface, but also change like the way it is with people.

So going from like, in a chat bot, to mobile collaborative agent, and the collaborator, it is like a step towards like more genetic systems that become like innovators, ultimately.

And so the entire team of like applied engineers, designers, products, like research kind of like got like formed in the air, almost out of like nothing, it's just like a collection of people who just like got together.

And the rapid list that is waiting for each other.

Actually, like canvas is like one of the, I would say like the first project at open AI where researchers and applied in here started working together from the very beginning of the product development cycle.

And I think like there's a lot of things that we have learned on the way.

But I definitely came to with the mindset of like, we need to do like a really rapid model situation, such that like it would be much easier for engineers to, you know, write with the latest model possible, but also learn from like, use a feedback or like early like internal dog foods, how to be improved the model very rapidly.

And, you know, it's really hard to like, kind of like figure out like how people when you deploy products, how people would be able to like use it.

And so like, the way you synthetically train the model is basically figuring out like, what are the most core behaviors that you want this product feature to do.

And for canvas, for example, it was, it came down to like three main behaviors, it was how do you trigger canvas for prompts like write me a long essay, when the user intention is mostly like iterating over long documents, or write me a piece of code, or when to not trigger canvas for prompts like, can you tell me more about precedent?

Like, I don't know.

So some of the general questions.

So you don't want to trigger canvas because the user intention is mostly getting answered and not necessarily like iterate over the long document.

The second behavior is how do you how do we teach the model to update the document when the user asks.

So one of the behaviors that the model is actually have like the some agency on autonomy to literally go to the document and like select specific sections and either delete it or edit so highlight it and rewrite certain sections.

So sometimes the model, sometimes the user would just like say, change the second paragraph to be something friendlier.

And you would have to like each the model to literally find the second paragraph in the document and change it to a friendly tone.

So basically, you teach both like how to trigger like, edit itself, but also how do you teach the model to get higher quality edit for the document.

In case of like coding, for example, there's also like the question of like, how good the model is like completely rewriting the document versus like having like very specific targeted edits.

So that's like another like layer of decision boundary within like edit itself.

It's like select the entire document that like rewrite completely, or you want to like have like very targeted custom behavior.

And you know, like when you first launch the model, we would bias the model towards like more rewrite, because we thought the quality of the rewrite were like much higher.

But over time, you're like kind of shifting based on like, you just see that and what's the learning from each other deployment.

Lastly, this the third behavior that we taught, and that actually, the model is how to make comments on any document.

So the way we use that is like, we would use a one model to produce to like simulate like use a conversation, let's say like, write me a document about x, y, z, but then we use a one to like produce the document.

And then we kind of inject it like user prompt to be like, oh, make some comments, critique my piece of writing, or critique this piece of writing that you just made.

And then we taught the model to like make comments on the document on like very specific documents, this is like, also like, what kind of comments you want the model to make?

Like, do they make sense or not?

Like, how do you teach the quality of that?

And it all came down to like measuring progress via very robust evolves.

But yeah, this is how you like use like a one like kind of synthetic data generation for like the screening.

Okay, that's so interesting.

So you talk about this idea of teaching the model and you mentioned how it's using synthetic data to teach the model different behaviors is a simple way to think about it.

Basically, that's where you do that by showing it what success looks like using basically evals.

Is that the simple way to think about it?

Like, here's what you doing this successfully would look like.

And that teaches it.

Okay, I see this is what I should do.

Great.

Yeah, amazing.

Yeah, you got it.

Okay, got it.

I want to start unpacking what your day to day looks like as you're building these sort of things.

Is it like you sitting there talking to some version of chat GPT, crafting these evals?

Sometimes I do that.

But sometimes I do sit.

I think I learned this so much from and that is like, people spend so much time just like problem models and like quality to a little bit all the time.

And you actually get a lot of new ideas.

How do you make the model better?

It's like, oh, like this is this response is kind of weird.

Like, why is it doing this?

And you start like debugging or something or like, you start like figuring out like new methods, like how do you teach the model to respond in a different way?

Like, how better personality, let's say.

So it's the same thing of like, how personality is made like in the models windows, like very similar methods.

But yes, I think my time, I don't know if I have changed, I think when I first came, I was like, mostly like research, I see work.

So I was like, building a lot of like, I was like, writing code, like, you know, changing models, writing evals, working with PMs and like designers to like, learn, teach them how to like, even think about like, innovations.

I think it was like, really cool experience.

And I think it was like an adoption of like, how do you like, do this like, prior management of like AI features or like, AI models?

Yeah, but now it's like, mostly like, you know, like management and like, mentorship.

I'm still like, doing SC like research code after like 4pm, although, but yeah, it's kind of like changed.

All right, don't talk too much about being a manager, because everyone's firing their managers who needs managers anymore.

That's the right here now.

Just kidding.

It's interesting that so much of your time was spent on teaching product teams how evals integrate and how important that is.

And I've heard this a few times, and I haven't personally experienced it yet.

So I think it's an important threat to follows just how writing these evaluations is going to become increasingly an important part of the job of product teams, especially when they're building AI features and working well.

So can you just talk a bit more about what that looks like?

Is it like sitting there with an Excel spreadsheet, basically showing like, here's the input, here's the output, here's how good the result was, talk about what that actually looks like very practically.

It certainly depends on the what you're developing.

But there are various types of like evaluations.

So sometimes I do ask product managers, or there's also like new role that we have like model designers to kind of like go through some of the user feedback, maybe you all like think of like various like user conversations that should have triggered like under this circumstances, it should trigger Canvas.

And then you have this like ground truth label of like, okay, with this conversation, it should look to go Canvas.

Under this conversation, it should not trigger Canvas.

And you have this like very bi deterministic kind of eval that for like, this is about it, behaviors is like this.

When we were launching tasks, for example, like, how do you make correct schedules is like actually really hard for the model.

But the view, we built out like some of the deterministic evaluations, that is like, okay, like if the user says like 7pm, it should like, the model should say 7pm.

So you can like have a deterministic evolves with those like paths or fail.

So yeah, and like the way it works is like, sometimes I ask, product managers just like go create like a goal sheet, like have different tabs and like, what's the current behavior?

What's like the ideal behavior?

And like why like some nodes, and sometimes we usually use it for eval, sometimes we use it for training.

Because like if you give this project to like a one model, it can probably figure out like how to teach itself a good behavior.

And I think there are second type of like evals, that is kind of more prevalent is like human evaluations.

And you can have specific trainers, or you can have like internal people to when you have like a conversation of the prompt, and then you have like, various completion of models, you kind of choose the win rate, which model is the best, which model produce the highest quality comment or edit, and then you can have like continuous win rates.

And as you develop new models, it should always like win over the previous models.

So it depends on what you want to measure.

It's so interesting.

Like basically, what I'm hearing and this is something I'm learning about as I talk to people is product development start might move from this like, here's a spec PRD, let's build it together.

And then cool, let's review it.

Are we happy with this too?

From that to, hey, AI, build this thing for me.

And here's what correct looks like.

And I'm spending all my time on what is correct look like on evals essentially.

You definitely want to like, measure progress, it's your model.

And this is where evals is because like, you can have prompted model as a baseline already.

And if the most robust evals is the one where prompted baselines get the lowest score or something.

And then because then you know, like, okay, if you're trained a good model, then it should like just like hill climb and that you all of the time, while not like also like regressing on like other intelligence evals.

So it's like, I think it's more what that's, that's what I'm saying, like it's more than art and science is like, okay, like if you optimize the model for this behavior, like you kind of don't want to like brain damage in like other areas of intelligence or this is happening like all the time in every lab and every like research team, I would say like prompting is like also a way to like prototype like new product ideas.

Like early days at Andorra Glune was working like file uploads feature.

I remember I was just like, you know, prompting the model to just like, I mean, when we were like launching like 100 key contexts, I was just like prototyping this in a local browser, which I did the demo, like people really, really loved it.

And they just like wanted like API for like file uploads or something.

And then that's when it clicked to me like, I also like wrote a blog post on table, like it clicked to me like prompting is like a new way of like product development or prototyping for designers and for like product managers.

For example, one of the features that I wanted to do is like have like personalized recommended personalized starter prompts.

So whenever you come to like cloud, like it should like recommend you like starter prompts based on what your interests are.

And so like you can literally do it like prompting for that.

And another feature was like generating titles for the conversations.

It's a very small like micro experience, but I'm really proud of the way we did that was because of the we took like five latest conversation from the user like as the model like what's the style of the user.

And then like for the next kind of new conversation, the generated title will be of the same like style.

The users like really little like micro experiences like this.

That's so cool.

Did you do that at the drop pick or at open AI?

I don't have it.

Okay, cool.

I love the file upload feature that Claude has, by the way, chat GPT doesn't have that yet.

Is that right?

I think it has.

I think like the way it's implemented is like very different though.

Okay, maybe it's the PDF feature because I use it all the time with call it okay, that's cool.

So I need to get on that.

Man, it's wild.

How many features you built that I use every day and that many people use every day.

This prototyping point you made is really important.

It's something that comes up a ton on this podcast also of how that is maybe the way that AI has most impacted the job of product builders recently is just prototyping instead of going from showing just like here's a PRD here is a design.

PMs more and more just here's the product type of the idea that I have and it's working you can play with it.

Yeah.

Yeah.

Okay, I want to spend a little more time on how you operate.

So you talked about you built this in launch of this tasks features that is that the way to describe your tasks.

Yeah.

So talk about how that emerged.

And let's better understand just how you collaborate with product teams and how open AI works in that way, whatever you can share there.

I think canvas and tasks are going into the bucket of objects where it's like more like short or medium terms.

And actually, the way canvas and tasks came up about to be was like, it started with like one person prototyping and creating like a spec.

It's kind of like PRD is like creating a spec of like the behavior of the model.

I don't think like tasks is like extremely groundbreaking, groundbreaking feature and necessarily what makes it like really cool is because the models are so general, model can now search, they can like write sci fi stories, they can like search for stocks, they can like summarize the news every day, because the models are so general, like giving something familiar to people that like, you know, notifications is like very familiar, like having reminders is like very familiar.

So like, feeling like a form factor for the people who like very familiar, same as like canvas, right?

It's like Google Docs is very familiar.

But then you add like magical AI moment and it becomes like very powerful.

But the way comes usually like operationally, like, yeah, it's like a prototype, like literally prompted prototype or like how you would want the model to behave.

For tasks, for example, like, you kind of like need to design a little bit like design systems design thinking is like, okay, like, well, is it more if this the user says, like, remind me to go to lunch, like at 8am tomorrow.

Okay, what what kind of information does the model needs to extract from that prompt in order to create a reminder.

And so this is how you like, like design like a stock for a new feature, like a tool, canvas and tasks, all tools.

So it's like, how do you like create the tool stock?

And then the site, mostly like, like, developing JSON schema was like, okay, like from this problem, maybe the model should extract like, the time that the user requested.

And then you think about like, what, which format do you want the time to be?

And then like, how do you want the model to like, not a fire is like, basically, if the user should give instruction to the model.

And then this instruction would like fire off like every day or something at that particular time.

So for example, if you say like, search, like every day I want to like learn know about the latest AI news, the models should rewrite into like, okay, like search for the latest AI news.

And this will will this task will get fired at that particular time that the model that the user requested.

And then you know, your design is like tools back.

And then actually, I don't know, like, I feel like sometimes like, it's like through conversations, I like, either like people asked me to like, join the team and they're like, Oh, my God, like we need to be search for this, or like, we need like some support, like, we need to train the models or something like this kind of was like, mostly like, I just pitched the idea of like, it got staffed quite immediately during the break.

So I know like, it's like the project.

And usually with staffing is like mostly like a product manager, model designer, actual product designer, a couple researchers like applied engineers, the puzzle, the complexity of projects.

And then like, you know, it took for tasks, it took like, I don't like two months or so to go from zero to one, basically.

For canvas is was like four or five months, I guess, to go from zero to one.

But yeah, then like, you know, you teach product managers how to like build the walls and like maybe, you know, how do we not only like ship the better feature, but how do we think like, more longer term, like, what kind of cool features that you want tasks to have?

Like, I think it would be nice for tasks to be like, extremely little bit more personalized, it'd be nice to have like, to create tasks via voice and on a mobile, right?

Like, so you kind of need to like, this is how you get like, we switch from up right now, right here is like, thinking like how the feature will be developed in the future.

And then from there, it's like, you like start creating data sets, like, with evos, you want to make sure that goes well, and then like, you need to have like a trade off between like what methods you want to use.

And the reason why I really love like, synthetic like relying purely on synthetic data instead of like collecting data from humans is because it's like much more scalable.

It's cheap, less than half like you literally sample from the model.

And you teach the core behaviors of the models and that will generalize on to all sorts of diverse coverage.

And when you launch the better feature, you learn so much from the users that you can like, all your synthetic sets can be shifted in the distribution of how the users behave and on the private behavior.

And this is how you improve.

And this is what happens kind of when you go from beta to J.

This episode is brought to you by Loom.

Loom lets you record your screen, your camera, and your voice to share video messages easily.

Record a Loom and send it out with just a link to gather feedback at context or share an update.

So now you can delete that novel length email that you were writing.

Instead, you can record your screen and share your message faster.

Loom can help you have fewer meetings and make the meetings that you do have much more productive.

Meetings start with everyone on the same page and end early.

Problem solved, time saved.

We know that everyone isn't a one take wonder when it comes to recording videos.

So Loom comes with easy editing and AI features to help you record once and get back to the work that counts.

Save time, align your team, stay connected, and get more done with Loom.

Now part of Atlassian, the makers of Jira.

Try Loom for free today at Loom.com/Lenny.

That's L-O-O-M.com/Lenny.

Something that I want to help people understand and I don't even 100% understand this is what's the simplest way to understand the job of a researcher versus say a model designer and other folks involved.

Like what's the simplest way to understand what researchers do at Open Air?

So the project that I described mostly like product oriented research is mostly a product research.

Another part component of my team is actually more like longer term exploratory projects.

And it's more about like developing new methods, understanding those methods, and a variety of circumstances.

So like basically developing methods, you kind of like need to follow very similar kind of like recipe of like building eviles.

But it's like more sophisticated eviles.

Like you kind of want to have like auto distribution or like if you want to like measure generalization, you kind of need to like capture that.

But it's basically more sciencey in a way where you know if we talk about synthetic data, like one of the hardest things about synthetic data is like how do you make it like more diverse?

Diversity in synthetic data is like one of the most important questions right now.

And it's like exploring like ways to inject like diversity as a general method that will work for all is like a one of the research explorations.

Other ones is like more like developing new capabilities.

I feel like it's always about like you know like you work on this new method and you have like signs of life that it's working.

Either you think of like how do you make it more general or you think of like how do you make it very useful or like and this is how like the longer term projects become more like medium like short term projects.

That makes sense.

Essentially working on developing ways to make the model smarter.

04 or 506.

Yeah.

Anyways to like 01 was a big breakthrough right the way it operates where it's not just here's your answer it actually thinks and has right takes time to think through the process of coming up with an answer.

Okay.

Yeah.

Very helpful.

Speaking of that I was thinking about the future where things are going.

I want to spend some time on just this insight that basically you are building the cutting edge of AI like at the very bleeding edge of where AI is going and where it is.

And so I'm very curious to hear just your take on how you think things are going to change in the world and how people work based on where you see things are going.

And I know it's a broad question but let's say like in the next three years how do you see the world changing how do you see people's way of working changing.

It's a very humbling experience to be in both labs I guess like to me when I first came to endowment I was like oh no I really love from that engineering and then like the reason why I switched to like research is because I realized at that time is like oh my god like cloud is getting better like front end.

Like cloud is getting better like coding.

I think cloud can like develop new apps or something and so like it can like develop new features for the thing that I'm working.

So it's like it was kind of like this meta realization where it's like oh my god like the world is actually changing and they're like when we first like launched a 100k context at that time obviously you know I'm thinking about like front factors that's like yeah like file uploads were like very natural very familiar to people but you can imagine we could just like make like infinite chats in the cloud.ai app right like as if like it's like a 100k context but because like file uploads it's like foreign follows function it's like the foreign factor the file uploads kind of enable people to just like literally upload anything the books or like any reports financial and like ask any task to the model and then I remember it was like you know enterprise customers like um like financial customers are like really interested in that as like oh wow like actually they it's actually one of the very common tasks that people do uh in that setting it was like kind of crazy to like see uh how some of the redundant tasks are getting like automated basically by these like smart models and they're entering the the era where I actually don't know for example sometimes like if l1 gives me the correct answer or not because I'm not an expert in that field and it's like I don't even know how to verify the outputs um are the models is because like all my experts know like they can like verify this so yes so basically there are trends that are going on the first trend is the cost of reasoning and intelligence is drastically going down I had a blog post about this maybe I should update on like latest benchmarks because at that time like mm everybody was like doing like um it's not like one benchmark and they'd be like quickly saturated the benchmarks and like now we need to like do the same flop but was with another like frontier eval but the cost of intelligence is like going down because it becomes like much cheaper smart small models are becoming even smarter than like large models and that's because of like the distillation research this happened with like claudity haiku I was like working on like pristine like claudity haiku and I realized it was much smarter than like claudity which was like way you know bigger lesson like that um but like the power of like small models become very intelligent and fast and cheap we are moving towards the world that has multiple implications but the news that like people will have more access to AI and that's really good like builders and developers will have much better access to AI but also it means like all the work that has been like bottlenecked by intelligence will be kind of like unblocked so anyone like I'm thinking about healthcare right like if I have instead of going to a doctor I can like ask chat GPT or give chat GPT a list of symptoms and ask me like oh which like would I have like a cold flu or like something else like I can literally get the access to like a doctor almost and there's like been some like research studies around that yeah there's a new york time story about that where they compared doctors to doctors using chat GPT to just chat GPT and just just chat GPT was the best of them all like jet like doctors made it worse yeah yeah that's crazy like right like education I think uh I will have friends if like I had the tool like chat GPT and when I was like young and like would learn so much but it's like people can now learn almost anything from these models so they can learn new language they can learn how to build new look apps like I don't know anything that you're wanting like I'm so like it's humbling to like have like launch canvas and like bring that thing to the people enable them to do something else that they couldn't have ever before I think this is there's something like magical around this experience as education has will have massive implications like I guess like scientific research right like I think it's like the dream of like any AI researchers like automate AI research uh it's kind of scary I'd say um which makes me think that like people management will stay you know it's like one of the hardest things to it's like emotional intelligence for the models or like creative creativity in itself is like one of the hardest things so writers I don't think like people should be worried as much I think it's like I think I'll alleviate a lot of like redundant tasks for people this is awesome okay I want to follow this thread for sure and it's funny that what you described is like you're an engineer entropic and you're like okay Claude is gonna be very good at engineering this isn't gonna be a potentially career long term so I'm gonna move into research and AI is gonna need me for a long time to build it to make it smarter I would say we still have like I think canvas team has still have like a really cool like front-end engineers that I really like you know people who like really care about like interaction design like interacting spirits like I don't think like models are there yeah it's like I think if but we can get the models to like this top one percent of like front end or something um for sure so what I want to move on to next along these lines is just and this is just speculation but uh what skills do you think will be most valuable going forward for product teams in particular so folks are listening and they're like okay this is scary what should I be building now to help me stay ahead and not be in trouble down the road what skills do you think are gonna be most more and more important to build yeah I think like creative thinking like you kind of want to like um come up with generate a bunch of ideas and like filter through them and they'll just like build the best product experience listening you know you want to like build something that like the most general model will not replace you and oftentimes you you build something and you make it really really good for like specific set of users and actually the mode is now in like your user feedback the mode is like more in like you whether you listen to them like whether you you can like rapidly iterate like the mode is like in here I don't think like we are yet to like there are so many ideas I feel in the abundance of like ideas that you can look at is like I wouldn't be worried I feel like in fact like I just think like people in AI fields are like I wish they were like a little more creative and like connecting dots across like different like fields or something like that to like develop really cool new like generation and new paradigms of interactions with this AI like I don't think we've cracked this problem at all um a couple years ago I was like telling some people I was like you know you kind of want to like build for the future so it's like it doesn't necessarily matter whether the model is good or not good right now but you can build product ideas such that like by the time the models will be really good it will work really well um I think it just like happened naturally like for example like I don't know like right like the Claude artifacts and I feel like early days of canvas was like back in like 2022 like before chai chai PT like writing ID was like kind of like chai chai but I feel like Claude 1.3 model itself was like not there to like made like really extreme good like high quality edits for example like coding um and I feel like I see like startups like Carcer and it's like doing super well like unless because you like iterate so fast they're like invent like new ways or like training models they move really fast they listen to like users like massive distributions like yeah it's capital that's really helpful actually so what I'm hearing is that soft skills essentially are going to be more and more important and powerful he's talked about management leading people being creative and coming up with innovative insights listening there's a post I wrote that I'll link to where I look I try to analyze what AI how AI will impact product management and we're actually very aligned and my sense was the same thing that soft skills are going to become more and more important and the things that are going to be replaced is the hard skills which is interesting because usually people value the hard skills like coding design writing really well and it's interesting that AI is actually really good at that because it's taking a bunch of data synthesizing it and writing creating a thing versus all these fuzzy things around of what influences convinces people to do things and aligning and listening like you said creativity anything along those lines come up as I say that I think it's actually really really hard to teach the model how to be aesthetic or like do like visual a really good like visual design or like how to be extremely creative in the way they write I think like I still think like GGP kind of sucks at like writing and that's because it's like it's like borrowed math by this like creative reasoning I think like prioritization is like one of the most important like I think like um for a manager I feel like I actually like AI research progress is all backed by like management like research management is because you have like this chain set of computes and you need to like allocate the computes to the research paths that you feel the most convinced about it was like you need to like really you need to have like a really high conviction and the research paths to put the compute and like it's more like return on investment kind of situation it's like okay yeah like I'm thinking a lot about like like okay like how do I across all my projects which projects are higher priority is like prioritization and also like on the lower levels like which experiments are really important to run right now and which are not and like cut through the line so I think like prioritization communication like um management um people's skills like empathy like understanding people like kind of like collaboration like I think like canvas wouldn't be like an amazing launch if it wasn't like about like people and I think it's a wonderful group of people and like I got a chance to like work with like people like Lee Byron who's like a co-creator like GraphQL and like some of the best like apple designers and it's like so cool to like see and like how do you create this like collaboration between people it's just like something that's still humane I think let me just follow through a little bit because I imagine people listening are like okay but once we have AGI or SGI it's like it'll do all this it's you know it's like there's a world where like why isn't all this done I think it's easy to just assume all that I'm curious this idea of creativity and listening why you think AI isn't good at it other than it's just very hard to train it to do this well is there anything there just like why this is especially difficult for AI now lamps to get good at I think currently it's difficult for many reasons I think it's still like not to like research area and it's something that like I think my team is like working on is like okay how like how do we teach the model to be like more creative and like the writing and actually like I thinking like doesn't your paradigm of life that the models think more should actually lead to like better writing in itself but like when it comes down to like idea generation or like um discriminating of like what is the good like visual design and art I feel like it hasn't had learned like examples from like people to discriminate it very well I do think it's because like you know there are not that many people who are like actually like really like oh it's not like accessible to like models so learn from these people I guess um so definitely that's why it sucks yeah that makes sense basically there's not enough of you yet uh researchers teaching it to do these things slash people that have incredible taste and creativity that can teach these things you could argue this will come but I'm not we don't need to keep going down that thread let me ask you a specific question in this post I wrote I made this argument that a lot of people disagreed with that strategy is something that AI tooling will become increasingly great at and take over there's the sense that that's the thing that people will continue to be much better at and you can't offload AI basically developing your strategy telling you what to do to win my case is isn't strategy you just take all the inputs all the data available understand the world around you and come up with a plan to win feels like AI would be like an LM would be incredibly smart at this what's your take I think so too I think like again like we you teach the model all sorts of like tools and like capabilities and like reasoning right and like when it comes down to like as for campus right now we've been very cool to like for the models just like aggregate all the feedback from users like summarize me like the top five like most painful flow flows or user experiences and then like the model itself is like very capable of like like thinking of like knowing how it's being made uh figure out like how to like create a data sets for itself to like train on it and I think like we are far away from that kind of like self-improvement models becoming like self-improved via like then like the product development is basically kind of like self-improving like it's kind of like it's all like organism or something um yeah like again like the strategy is like it's more like data analysis and like um coming up with like like I think what models are really good is like um we're connecting the dots I think it's like okay if you have user feedback from this source but you also have an internal like dashboard with metrics and then you have you know like other kind of like feedback um or like inputs and then like it can co-create like a plan for you like recommendations event and I think this is like one of the most common use cases for trashy pd2 is like coming up with like these sorts of things that makes sense like essentially a human can only comprehend so much information at once and look at so much data at once to synthesize takeaways and as you said these context windows are huge now here's all the information what's the most important thing I should do yes it was like scientific research is because like you like ideally the model would be able to like suggest like ideas like new ideas or like iterate on the experimental like given the empirical results of the previous experiments like how do you like come up with like new ideas or like the methods yeah oh man uh okay so just to close the loop on this conversation this part of the thread is the skills you're suggesting people focus on building and leaning into soft skills like creativity managing influence collaboration looking for patterns is that generally where your mind is at yeah I'm thinking a lot about like how do we make our relations more effectively anything that is mostly like management I guess it's like how do you organize like research teams or like generally teams like combine a composed team such that they will be at their maximally succeed or like at the maximum like performance of what can possibly like we can like literally create like the next generation of computers it's just like the matter of conviction and like the way you manage through that it's like scaling organizations or like scaling product researchers yeah I think what like you're basically building this thing and not efficiently doing it is like limiting the potential of the human species right now is right mismanagement within the research team and open i an anthropic and some of these other models yeah it's kind of crazy to think about holy moly okay so speaking of anthropic and open AI you've worked at both very few people have worked at both companies and seen how they operate i'm curious just what you've noticed about the differences between these two how they operate how they think how they approach stuff what can you share along those lines it's more similar than different uh obviously there is a lot of like there are some like differences also comes to like nuances uh to culture i really love on topic and i have a lot of friends there and i also love opening AI and i still have a lot of friends though so it's like it's not about like enemy i feel like there's like in the eye was all like yeah the competitors those like enemies this is actually like one big community and like as people like doing the same thing i would say what i've learned from on topic is this like real care and craft towards like model behavior model cost model training and i've been thinking a lot about like okay like what makes cloud cloud and what makes chat chat chat chat and it's like i tell it comes down to like operational processes that kind of leads to the outputs to to the model uh is the output of model and it's like the reason why collide has so much more personality and like uh it's more like a librarian i don't know like i don't know i'm like visualizing a cloud being like a librarian like a um very like nerdy or something um it's because i feel like it's a reflection of the creators who like making this model and like a lot of like details around like the character and the personality and like whether the model should follow up on this question or like not like was the correct like ethical behavior for the model to like in this scenario it's like a lot of like crafts um and like true beauty like the assets and this is where i learned that part of like art i guess uh i don't know but i'd say like antarborist like much smaller like when i joined it was like what like so many people when i last was a kind of people and like obviously the culture changed so much i really enjoyed being like early days startup like lives and like people knew each other as a family but like the culture shifted i would say like and i learned from antarborist that like they're much better at like focusing and like prioritization like very very hard like very hardcore prioritization i guess and they need to do it like but i think like open the eyes like much more um innovative and uh much more like risk takers in terms of like product or like research actually you know like i don't know you can like your full-time job can be just like teaching the model how to be like creative writers and it's like there's some luxury in this like research freedom that that comes to scale maybe i don't know um but it gives you it's like you'll have i feel like i have much more creative like product freedom to do almost anything i guess within like opening eye like a raw strategy into like the vision that you want it's like more like yeah make probably bottoms up i guess yeah that's how i was thinking about it it feels like opening eyes more bottoms up uh distributed people bubble up ideas try stuff there's more and that leads to more products launching i imagine more things just kind of being tried versus more of a let's just make sure everything we do is awesome great and craft and thinking deeply about every investment that's really interesting i've never heard it described this way uh carina we've covered so much ground this is going to help a lot of people with so many uh ways of thinking about where the feature is going before we get to our very exciting lighting around i'm curious if there's anything else that you think might be helpful to share or get into one of my regrets i guess when i was early days around ever was that like i think there was like some luxury of the time this pre-chai tp key to actually like come in with like a bunch of ideas and like prototype like almost every day um and i think like we did a lot of cool ideas like clod and slack was actually one of the first like uh tool uz like products is like a clod could operate in like your workplace now it's like kind of cool when you like add clod to summarize the thread so maybe you have a entire conversation with someone and then you want to like a summary or like what happened like you can say like add clod summarizes also it was really fun to like even like iterate on the model itself it's like when you just like talk to the model and like slack forever um if creators don't get some social element it's kind of always kind of let me join me and like um this discord like people learned so much about like prompting and like how do you work with like clods actually one of the features that was like early tasks prototype was like you know every monday clod would just like summarize the entire channel or like every friday we just like summarized like a bunch of channels and give like the news about the organization or something so it's kind of like really cool like phone factor i think i'm thinking about like phone factors like a really important question like in ai especially we haven't like even figure ads like how do we create like an awesome like product experience with like oh serious models it's like the paradigm between like synchronous real time give an answer paradigm into like more asynchronous paradigm of like agents working on the background but then now the question is like the agents should both trust with you right and trust both over time which is like with humans and um you know you saw this collaboration which is why like it collaborate like this collaboration model was like you and the models like so important because you both trust and the model learns from preferences so that it can become like more personalized and it will start predicting the next like action that you want to take on the computer or something and it's like kind of like more predictive much more we went from like personal computers like personal model uh basically here yeah that's uh why is it not a thing that seems like such an obvious feature that every lm should have is a slackpot version of them is that is that a thing i can have you install or is that not a thing right now i know that cloud and slide was sunsetted in like 20 23 or something but i think i think uh i think it was like after chai chippy it was mostly like the focus on like consumer use cases or like enterprise use cases uh i think i think the form factor of like cloud and slack is like um was kind of constrained a little bit uh when you want to develop new features i want that i know that chai chippy had like slack part too so i don't know like maybe it will come back all right i would i would pay for that uh any other memories from that time of early days because that's a really special place to have been as early days anthropic any other memories or stories from that time that might be interesting to share i think the very first launch when they felt like when clips and use again was like a hundred key context um launch is like when the models could input the entire like book and give you like summary of the book or something um or the entire financial or like have like multi-files financial reports and then like give you an answer um to the question to very specific question i think there was something in there that kind of like oh my god this is like a really cool new capability not like model capability but more like the capabilities that came from the product form factor itself rather than like the model capability as much um i think like other prototypes that we were thinking about like yeah like there's like one part of a cloud workspaces and it's like kind of same like idea like cloud and i would have to share workspace and that share workspace like a document and we can like it to it in the document i feel like sometimes the ideas like private bs lag and they lag for like two years um just like in this case it's interesting there's these milestones i kind of uh open up our view of what is happening and where things are going chat gpt i think was the first of just like wow this is much better than i would have thought you talked about 100k context windows where you could upload a book and ask you questions have it summarized i actually use that all time when i have interview guests and they wrote a book i sometimes don't have time to read the whole book so i use it to help me understand what the most interesting parts are and then i actually dive into the book just to be clear uh and then i don't know maybe like voice was another one where you could talk to say chat gpt is there any other moments there that you're like wow this is much better than i thought it was gonna be yeah i think like uh the computer use agents like the model operating the desktop and you can essentially think of like you know new kind of like experience where the model can learn the way you browse and from that preference it can just like browse as just like you and it's kind of like simulation simulated persona and it's actually very similar to the idea of like okay like maybe sam almond doesn't have a lot of like um time maybe i want to like talk to like his simulated like his simulation and ask like oh like for example like yeah like i i really appreciate some of the technical mentorship like yeah cool like but he doesn't have a lot of time so it's like i really want to like ask him these questions like how you would respond like simulated environments like this um would be really cool that's a great place to plug lenny bot i have one of those it's trained on all of my podcast and newsletters and i it sits on many models i don't know which one exactly they use but it's exactly that it's uh and it's not even me it's all the guests that have been on the podcast on these letters i wrote and you could just ask it how do i grow my product how do i develop a strategy and it's actually shockingly good do you feel like it reflects who you are politically okay the best part of it is you can talk to it it's built there's an 11 labs voice version that's trained on my voice non-fitness podcast and it's actually very good and people like have told me they sit there for hours talking to it wow and somebody uh told it interview me like i am on lenny's podcast ask me questions about my career and he did a half hour podcast episode with lenny that's so fun it's incredible future is wild yeah i think like content transformation is like you know like i would imagine sometime like you know um when you generate a sci-fi story in canvas like you can like transform this into like audio blog like where you have like very natural like content transformation blog around media to another medium i think like one of my earliest inspiration is like one of the last episodes of like west world where uh i don't know but where dolores comes to her work at the time and she comes to like this like new workspace and she starts like writing a story and then she writes a story like a 3d like virtual reality starts like creating on the fly so i don't want to read that um kind of cool wow speaking of medium uh uh i guess i i was wondering if i should go in the structure now but real quick uh kevin weil slash kevin wheel i don't know exactly how to pronounce his last name the cpo of uh is a while or wheel i think real wheel okay okay let's just say that okay we open up he was uh he did a panel at the lenient friend summit last year and he made this really fascinating point that chat is a really interesting interface for these tools because they're just getting smarter and smarter smart and smart and smarter and chat continues to work as a paradigm to just interact with them similar to human you could talk to albert einstein you could talk to someone not very smart and it's all conversation still and so it's a really flexible way to interact with increasingly good intelligence at some point it'll not be so great and you're talking about all these ways that you're adding additional ways to interact but it's interesting chat proved to be a really powerful layer on top of all the stuff yeah that's really cool i feel like child also has the social element which is like very uh actually it's like you know you sometimes want to like get into group chat and like yeah having conversations with the eyes kind of like a group chat in itself it's like massive thing actually this this idea of like how do you build like features like this like i see tasks as like this like um general kind of like feature that will scale very nicely as the models would develop like new capabilities ourselves it's like like the model will be able to like do better like searches and like you know create new like come up with like more creative like writing and like render you know react apps and like html preview like ops and like you can have like every day a new puzzle for you like every day like continue the story from the future it's like it scales very nicely you mentioned something as we were getting into this extra section that we ended up going down is this idea of uh your the agents using a computer i know this is actually something you are going to launch today the day we're recording it which will be out by the time this comes out call operator can you talk about this very cool feature that people will have access to yeah so uh i unfortunately did not work on that but i'm really really excited about like this launch um it's basically an agent that can complete the task in its own like virtual computer like in its own virtual environment you can do any literally tasks like ordering me a book on amazon and then ideally the model will either like follow up with you like which book do you want or like know you so well they will like start recommending like oh here's the five books that i might recommend to you to buy and then like you hit like yeah help me help me buy and then uh the model goes off uh into its own like virtual little browser and like complete the task and buy the book in the amazon and then if you give the model like credentials kind of cards obviously it comes with like a lot of trust and like safety um then it will just complete the thing for you it's a virtual assistant it's interesting how this just sounds like obviously this should happen like why is this not a yeta thing which is also mind-blowing that we're just assuming this should exist like just some ai doing things for you on a computer you just ask it to do like it's absurd it's actually really hard and i think like um you're just still cracking this but i feel like i don't know if you use like tuple it's like a pair programming product nope but um i don't know if you love pair programming so if you use shopify uses this i remember came up on a podcast episode oh nice yeah so it's a very cool product where you can just like call anyone at any time and then like share screen and the other person can like have access to the stream or like start like literally operating your computer and it's very like real-time like the allegiance is like very um it's like very high quality um and it's just like i kind of want the same it's like i want to like pair program with like my model and like the model should even talk to me like draw like very specific like section in my code and yes go to like tell me like i will teach me and you can have like different modes it's like right here it's like a product right here for you i don't know um people some people should build build that it sounds like a startup just got birthed yes from someone listening to this you mentioned that it's very hard to do this uh agent controlling a computer as you and helping out what makes it so hard for whatever however much you can explain briefly much of it is like uh because right now the models operating on like pixels instead of like language or whatnot like pixels is actually really really hard models because like perceptual visual perception i think there's a still like a lot of like multimodal like research that's going on um but i think like language scales so much like easier compared to like multimodals because of that another like thing that i just like my team is working on is like how do you derive human intent um very correctly it's like sometimes like does the model know enough information to ask a follow-up question or like to complete the task you kind of don't want like an agent to like go off for like 10 minutes and then compile with like an answer that you didn't even want that actually creates like much more verse user experience and this is comes with like teaching the model like like people skills it's like you know like what do people like like kind of like creating like the mental model of the user and like care about the user in order to ask certain questions like actually that part is like hard to for the models that relates to what we talked about earlier where this kind of the soft skill people skills pieces not where these models are strong yet okay i'm gonna skip the lightning round i want to ask just one question from lightning round something fun yes uh okay so when ai replaces your job karina i'm curious what you're and it gives you a stipend gives you a monthly stipend here's your here's your salary for the month what would you want to do what do you want to spend your time on what will you be doing in this future world i've been thinking about this all the times i have i feel like i have a lot of jobs options i would love to be a writer i think i think that would be super cool um just to like write like short stories like sci-fi stories um novels i really like art history so you know it's like um conservationists so like in the museums who just like try to preserve like art paintings but just like painting through a lot of things i feel that would be really cool um to do um yeah that sounds beautiful i don't know uh what i'm hearing is you need to nerf these models to not get very good at writing so that you can continue although at that point you don't need to do it from like you don't need people to buy it you're just doing it for fun so it doesn't even matter if they're incredibly good at writing or art conservation oh man what an episode or conversation what a wild time we're living in carina thank you so much for being here two final questions where can folks find you online if they want to reach out and follow up on anything and how can listeners be useful to you you can sign me i'm on twitter it's kenemian um you can also shoot me an email on my website um and i'm my team is hiring and so like i'm looking for research engineers research scientists as well as like machine learning engineers like people who come from like product engineers who want to like learn like model training um i'm actually hiring for like my team my team is called like frontier product research and the train models we develop new methods but for product-oriented outcomes what a place to work holy moly uh what's the best way for people to apply for these uh very lucrative roles i think you can shoot me a dm twitter okay or um i'm yet to create a job description okay this is the job description are you gonna apply under like post training team yeah okay so you're gonna get a flood of dms i hope you're prepared karina thank you so much for being here this was incredible thank you so much lenny bye everyone thank you so much for listening if you found this valuable you can subscribe to the show on apple podcasts spotify or your favorite podcast app also please consider giving us a rating or leaving a review as that really helps other listeners find the podcast you can find all past episodes or learn more about the show at lenny's podcast.com see you in the next episode