Dario Amodei on AI Progress, Scaling, and Governance

Source material

Transcript

So we talked three years ago.

I'm curious in your view.

What has been the biggest update of the last three hours?

What has been the biggest difference between what I felt like last three years versus now?

Yeah, I would say actually the underlying technology like the exponential of the technology has has gone Broadly speaking, I would say about about as I expected it to go.

I mean, there's like plus or minus, you know, a couple.

There's plus or minus a year or two here.

There's plus or minus a year or two there.

I don't know that I was predicted the specific direction of code.

But actually when I look at the exponential, it is roughly what I expected in terms of the March or the models from like, you know, Smart High School student, a smart college student to like, you know, beginning to do PhD and professional stuff and in the case of code reaching beyond that.

So, you know, the frontier is a little bit uneven.

It's roughly what I expected.

I will tell you though what the most surprising thing has been.

The most surprising thing has been the lack of public recognition of how close we are to the end of the exponential.

To me, it is absolutely wild that, you know, you have people, you know, within the bubble and outside the bubble, you know, But you have people talking about these these, you know, just the same tired old hot button political issues and like, you know, Or around us, we're like, here's the end of the exponential.

I want to understand what that exponential looks like right now because the first question I asked you when we recorded three years ago was, you know, what's up at scaling, how it might as a work.

And I have a similar question now, but I feel like it's a more complicated question because at least from the public's point of view, yes, three years ago there were these, you know, Well, known public trends were across many of us in magnitude of compute, you can see how the loss improves.

And now we have RL scaling and there's no publicly known scaling law for it.

It's not even clear what exactly the story is of is this supposed to be teaching the model skills, it's supposed to be teaching metal learning.

What is the scaling hypothesis at this point?

Yeah, so, so I have actually the same hypothesis that I had, even all the way back in 2017.

So in 2017, I think I talked about it last time, but I wrote a doc called the big blob of compute hypothesis.

And you know, it wasn't about the scaling of language models in particular.

When I wrote it, GPT1 had just come out, right?

So that was, you know, one among many things, right?

There was back in those days, there was robotics, people tried to work on reasoning as a separate thing from language models.

There was scaling of the kind of RL that happened, you know, kind of happened in AlphaGo, and, you know, that that happened at Dodat, OpenAI, and, you know, people remember Starcraft at DeepMind, you know, the AlphaStar.

So it was written as a more general document.

And the specific thing I said was the following that, and, you know, it's very, you know, starting to put out the bit or less in a couple years later, but, you know, the hypothesis is basically the same.

So, so what it says is, all the cleverness, all the techniques, all the kind of, we need a new method to do something like that, doesn't matter very much.

There are only a few things that matter, and I think I listed seven of them.

One is like, how much raw compute you have.

The other is the quantity of data that you have.

Then the third is kind of the quality and distribution of data, right?

It needs to be a broad, broad distribution of data.

The fourth is, I think, how long you train for.

The fifth is, you need an objective function that can scale to the moon.

So the pre-training objective function is one such objective function, right?

Another objective function is, you know, the kind of RL objective function that says, like you have a goal, you're going to go out and reach the goal.

Within that, of course, there's objective rewards, like, you know, like you see in math and coding, and there's more subjective rewards like you see in RL from human feedback or kind of higher order of versions of that.

And then the sixth and seventh were things around kind of like normalization or conditioning, like, you know, just getting the numerical stability, so that kind of the big blob of compute flows in this laminar way, instead of, instead of running into problems.

So that was the hypothesis and it's hypothesis, I still hold.

I don't think I've seen very much that is not in line with that hypothesis.

And so the pre-trained scaling laws were one example of kind of what we see there.

And indeed, those have continued going.

Like, you know, you know, I think now it's been, it's been widely reported, like, you know, we feel good about pre-training.

Like, pre-training is continuing to give us gains.

What has changed is that now, we're also seeing the same thing for RL, right?

So we're seeing a pre-training phase, and then we're seeing like an RL phase on top of that.

And with RL, it's actually just the same, like, you know, even even other companies have published, like, you know, in some of their, in some of their releases have published things that say, look, you know, we train the model on math contests, you know, AIME or the kind of other things.

And, you know, how well the model does is log linear and how long we've trained it.

And we see that as well.

And it's not just math contests.

It's a wide variety of RL tasks.

And so we're seeing the same scaling in RL that we saw for pre-training.

And you mentioned which are sudden and a bit of a lesson.

Yeah, I interviewed him last year.

And he is actually very non-LL unpilled.

And if I, I don't know if this is perspective, but one way to paraphrase this objection is something like, look, something which possesses the true core of human learning would not require all these billions of dollars of data and compute and these bespoke environments to learn how to use Excel or how to use PowerPoint and how to navigate a web browser.

And the fact that we have to build in these skills using these RL environments hints that we're actually lacking this core human learning algorithm.

And so we're scaling the wrong thing.

And so yeah, that is a great question.

Why are we doing all this RL scaling if we do think there's something that's going to be human like an disability to learn on the fly?

Yeah, yeah.

So I think this kind of puts together several things that should be kind of thought of thought of differently.

Yeah.

I think there is a genuine puzzle here, but it may not matter.

In fact, I would guess it probably, it probably doesn't matter.

So let's take the RL out of it for a second.

Because I actually think RL and it's a red hearing to say that RL was any different from pre-training in this matter.

So if we look at pre-training scaling, it was very interesting back in, you know, 2017 when Alec Radford was doing GPT-1.

If you look at the models before GPT-1, they were trained on these data sets that didn't represent a wide, you know, distribution of text, right?

You had like, you know, these very standard, you know, kind of language modeling benchmarks.

And GPT-1 itself was trained on a bunch of, I think it was fanfiction actually.

But, you know, it was like literary text, which is a very small fraction of the text that you get.

And what we found with that, you know, and in those days it was like a billion words or something.

So small data sets and represented a pretty narrow distribution, right?

Like a narrow distribution of kind of what you can see, what you can see in the world.

And it didn't generalize well.

If you did better on, you know, the, you know, if I forgot what it was, some kind of fanfiction corpus, it wouldn't generalize that well to kind of the other tab.

You know, we had all these measures of like, you know, how well does the, how well does the model do it predicting all of these other kinds of text.

You really didn't see the generalization.

It was only when you trained over all the tasks on the, you know, the internet when you kind of did a general internet scrape, right, from something like, you know, common crawl or scraping links and read it, which is what we did for GPT-2.

It's only when you do that that you kind of started to get generalization.

And I think we're seeing the same thing on RL that we're starting with first, very simple RL tasks like training on math competitions.

Then we're kind of moving to, you know, kind of broader, broader training that involves things like code as a task.

And now we're moving to do kind of many, many other tasks.

And then I think we're going to increasingly get generalization.

So that, that kind of takes out the RL versus the pre-training side of it.

But I think there is a puzzle here either way, which is that on pre-training.

When we train the model on pre-training, you know, we use like trillions of tokens, right?

And humans don't see trillions of words.

So there is an actual sample efficiency difference here.

There is actually something different that's, that's happening here, which is that the model start from scratch.

And, you know, they have to get much more, much more training.

But we also see that once they're trained, if we give them a long context length, they only think blocking a long context length is like inference.

But if we give them like a context length of a million, they're very good at learning and adapting within that context length.

And so I don't know the full answer to this.

But I think there's something going on that pre-training, it's not like the process of humans learning.

It's somewhere between the process of humans learning and the process of human evolution.

It's like it's somewhere between, like, we get many of our priors from evolution.

Our brain isn't just a blank slate, right?

The book's been written about.

I think the language models, they're much more blank slate.

They literally start as like random weights.

Whereas the human brain starts with all these regions.

It's connected to all these inputs and outputs.

And so maybe we should think of pre-training and for that matter, RL as well, as being something that exists in the middle space between human evolution and, you know, kind of human on the spot learning.

And as the in context learning that the models do as something between long-term human learning and short-term human learning.

So, you know, there's this hierarchy of like there's evolution, there's long-term learning, there's short-term learning, and there's just human reaction.

The LOM phases exist along the spectrum, but not necessarily exactly at the same points.

There's no analog to some of the human modes of learning that LOMs are kind of falling between the points.

Does that make sense?

Yes.

Some things are still a bit confusing.

For example, if the analogy is that this is like evolution, so it's fine that it's not that simple efficient.

Then like, well, if we're going to get the kind of super simple efficient agent from in context learning, why are we bothering to build in, you know, there's our environment companies which are, it seems like what they're doing is they're treating it.

How to use this API, how to use Slack, how to use whatever.

It's confusing to me why there's so much emphasis on that.

If the kind of agent that can just learn on the fly is emerging, where it's going to soon emerge, or has already, So I mean, I can't speak for the emphasis of anyone else.

I can only talk about how we think about it.

I think the way we think about it is the goal is not to teach the model every possible skill within RL, just as we don't do that within pre-training, right?

Within pre-training, we're not trying to expose the model to, you know, every possible, you know, way that words could be put together, right?

You know, where it's, it's rather that the model trains on a lot of things and then, and then it reaches generalization across pre-training, right?

That was, that was the transition from GPT1 to GPT2 that I saw up close, which is like, you know, the model reaches a point.

You know, I like had these moments where I was like, oh, yeah, you just give the model like, you just give the model a list of numbers that's like, you know, You know, this is the cost of the house, this is the square feet of the house, and the model completes the pattern and does linear regression, like not great, but it does it, but it's never seen that exact thing before.

And, and so, you know, to the extent that we are building these RL environments.

The goal is very similar to what is, you know, to what was done five or ten years ago with pre-training, with we're trying to get a whole bunch of data, Not because we want to cover a specific document or a specific skill, but because we want to generalize.

I mean, I think the framework you're laying down obviously makes sense, like we're making progress for a CGI.

I think the crux is something like, nobody at this point disagrees that we're going to achieve a CGI in the century.

And the crux is, you say we're hitting the end of the exponential, and somebody else looks at this and says, oh, yeah, we're making progress.

We've been making progress since 2012, and then 2035 will have a human-like agent.

And so I want to understand what it is that you're seeing, which makes you think, yeah, obviously we're seeing the kinds of things that evolution did, or that human, but then human lifetime learning is like in these models.

And why think that it's one year away and not ten years away?

Yeah, I actually think of it as like two, there's kind of two cases to be made here, right?

Two claims you could make, one of which is like stronger and the other of which is weaker.

So I think starting with the weaker claim, you know, when I first saw the scaling back in like, you know, 2019, you know, I wasn't sure.

You know, this was the whole, this was kind of a 50-50 thing, right?

I thought I saw something that was, you know, and my claim was this is much more likely than anyone thinks it is.

Like this is wild, no one else would even consider this.

Maybe there's a 50% chance this happens.

On the basic hypothesis of, you know, as you put it within ten years, we'll get to, you know, you know, what I call kind of country of geniuses at a data center, I'm at like 90% on that.

It's hard to go much higher than 90% because the world is so unpredictable.

Maybe the irreducible uncertainty would be if we were at 95% where you get to things like, I don't know, maybe multiple companies have, you know, kind of internal turmoil, and nothing happens, and then Taiwan gets invaded and like all the, all the fabs get blown up by missiles and, and, you know, and that now you would make the scenario.

Yeah, you know, just, you could construct a scenario where there's like a 5% chance that it, you know, you can construct a 5% world where like things, things get delayed for, for, for, for, for, for 10 years.

That's maybe 5%.

There's another 5% which is that I'm very confident on tasks that can be verified.

So I think I think with coding, I'm just except for that irreducible uncertainty.

There's just, there's, I mean, I think it'll be there in one or two years.

There's no way we will not be there in 10 years in terms of being able to do it end to end coding.

My one little bit, the one little bit of, of, of fundamental uncertainty, even on long time scales, is this thing about tasks that aren't verifiable, like planning a mission to Mars, like, you know, doing some fundamental scientific discovery, like, like CRISPR, like, you know, writing a writing a novel, hard to, hard to verify those tasks.

I am almost certain that we have a reliable path to get there.

But like, if there was a little bit uncertainty, it's there.

So, so, so, so, so, so on the 10 years, I'm like, you know, 90% which is about a certain as you can be.

Like, I think it's, I think it's crazy to say that this won't happen by, by, by 2035.

Like, in some sane world, it would be outside the mainstream.

But, but the emphasis on verification, hints to me as a lack of, a lack of, uh, belief that these models are generalized, if you think about humans, you're better things that both, which we get, a verifiable reward, and things which we don't, you're like, you have to start.

The, the, the, no, no, no, this is this is why I'm almost sure.

We already see substantial generalization from things that, that verify to things that don't, we're already seeing it.

But, but it seems like you were emphasizing this as a spectrum, which will, uh, split apart, which, you know, you see more progress, and I'm like, but that's, it doesn't seem like how it's getting, but the world in which we don't make it, or, or the world in which we don't get there is the world in which we do, we do all the things that are, that are verifiable, and then they, like, you know, many of them generalize, but what we kind of don't get fully there.

We don't, we don't, we don't fully, you know, we don't fully color in this side of the box.

It's, it's, it's, it's not a, it's not a binary thing, but, but it also seems to me, it's, even, even if, even if it's in the world, or generalization is weak when you only say it, or verifiable domains, it's not clear to me in such a world, you could automate software engineering, because software, like, in some sense, you are, quote unquote, a software engineer.

Yeah, but you have part of being a software engineer for you involves writing these, like, long memos about your grand vision about different things.

And so I don't think that's part of the job of sweet.

That's part of the job of the company, but I do think sweet involves, like, design documents and other things like that.

Which, by the way, the models are not bad.

They're already pretty good at writing comments, and so with, again, I, again, I'm making, like, much weaker claims here than I believe, to, like, you know, to, to kind of set up a, you know, to distinguish between two things, like, we're, we're already almost there for software engineering.

And we are already almost there by, by what metric, there's one metric, which is like, how many lines of code are written by A and if you use, if you consider other productivity improvements in the course of the history of software engineering, compilers write all the lines of software.

And, but we, there's a difference between how many lines are written, and how big the productivity improvement is.

Oh yeah.

And then, like, we're almost there meeting, like, how big is the productivity improvement?

Not just how many lines are written.

Yeah, yeah.

So, so I actually, I actually agree with you on this.

So I, I've made this series of predictions on code and software engineering, and, and, and I think people have repeatedly kind of misunderstood them.

So, so let me, let me, let me, let me lay out the spectrum, right?

Like, I think it was like, you know, like, you know, eight or nine months ago or something, I said, you know, they, I model will be writing 90, 90% of the lines of code in, like, you know, three to six months, which, which happened at least at some places, right?

Happened to happen to at and drop it, happened with many people downstream using our models.

But, but that's actually a very weak criterion, right?

People thought I was saying, like, we won't need 90% of the software engineers.

Those things are worlds apart, right?

Like, I would put the spectrum as 90% of code is written by the model, 100% of code is written by the model, and that's a big difference in productivity.

90% of the end to end sweet tasks, right?

Including things like compiling, including things like setting up clusters and environments, testing features, writing memos, 90% of the sweet tasks are written by the models.

And even when that happens, doesn't mean software engineers are out of a job, like, there's like new higher level things they can do, right?

They can manage.

And then there's a further down the spectrum, like, you know, there's 90% less demand for trees, which I think will happen, but like, this, this is a spectrum.

And, you know, I wrote about it in the adolescence of technology where I went through this kind of spectrum with farming.

And so I actually totally agree with you on that.

It's just, these are very different benchmarks from each other, but we're proceeding through them super fast.

It seems like in part of your vision, it's like going from 90 to 100.

First, it's going to happen fast.

And two, that somehow that leads to huge productivity improvements.

Whereas when I noticed, even in green field products, if people start with cloud code or something, people report starting a lot of projects.

And I'm like, do we see in the world out there of Renaissance of software, all these new features that wouldn't exist otherwise?

And at least so far, it doesn't seem like we see that.

And so that doesn't make me wonder, even if like I never had to intervene on cloud code.

There is this thing, like, there's just the world that's complicated, jobs are complicated, and closing the loop on self-contained systems, whether it's just writing software or something, how much broader gains we would see just from that.

Maybe that makes this a damn good estimation of the country of geniuses.

Well, actually, I like simultaneously, I simultaneously agree with you, agree that it's a reason why these things don't happen instantly.

But at the same time, I think the effect is going to be very fast.

So like, I don't know, you could have these two poles, right?

One is like, you know, AI is like, you know, it's not going to make progress, it's slow, like it's going to take, you know, kind of forever to diffuse within the economy, right?

Economic diffusion has become one of these buzzwords that's like, a reason why we're not going to make AI progress, or why AI progress doesn't matter.

And you know, the other axis is like, we'll get recursive self-improvement, you know, the whole thing, you know, can't you just draw an exponential line on the curve, you know, it's, you know, we're going to have, you know, Dyson spheres around the sun, and like, you know, so many nanoseconds after, you know, after after we get recursive.

I mean, I'm completely caricaturing the view here, but like, you know, they're, there are these two extremes, but what we've seen from, from the beginning, you know, at least if you look within anthropic, there's this bizarre 10x per year growth and revenue that we've seen, right?

So, you know, in 2023 it was like 0 to 100 million.

2024 it was 100 million to a billion.

2025 it was a billion to like 9 or 10 billion.

And then yesterday just about like a billion dollars with your own products, so you could just like, have a clean 10V.

And the first month of this year, like that, that exponential is, you would think it would slow down, but it would like, you know, we added another few billion to like, you know, to, to, we added another few billion to revenue in January.

And, and so, you know, obviously that curve can't go on forever, right?

You know, the GDP is only so large.

I don't, you know, I would even guess that it bends, bends, bends somewhat this year.

But like, that is like a fast curve, right?

That's like a, that's like a really fast curve.

And I would bet it stays pretty fast even as the scale goes to the entire economy.

So, like, I think we should be thinking about this middle world where things are like extremely fast, but not instant, where they take time because of economic diffusion, because of the need to close the loop because, you know, it's like this fiddly, oh man, I have to do change management within my enterprise.

You know, I have to like, you know, you know, I, I like, I set this up, but you know, I have to change the security permissions on this in order to make it actually work.

Or, you know, I had this like old piece of software that, you know, that like, you know, checks the model before it's compiled and, and, and, like, released, and I have to rewrite it.

And yes, the model can do that, but I have to tell the model to do that.

It has to take time to do that.

And, and, and so I think everything we've seen so far is, is compatible with the idea that there's one fast exponential that's the capability of the model.

And then there's another fast exponential that's downstream of that, which is the diffusion of the model into the economy.

Not instant, not slow, much faster than any previous technology, but it has its limits.

And, and, and, and this is what we, you know, when I look inside andthropic, when I look at our customers, fast adoption, but not infinitely fast.

Can I try a hot take on you?

Yeah, I feel like diffusion is cope that people use to say when it's like, if the model isn't able to do something, they're like, oh, but the diffusion issue, but then you should use the comparison to humans.

You would think that the inherent advantages that AI's have would make, diffusion a much easier problem for new AI's getting onboarded than new humans getting onboarded.

So, and AI can read your entire slack and your drive in minutes.

They can share all the knowledge that the other copy, other copies of the same instance have.

You don't have this adverse selection problem when you're hiring AI's, who's going to just hire copies of a VEDDI model.

Hiring a human is like so much more hassle.

And people hire humans all the time, right?

We pay humans upwards of $50 trillion in wages, because they're useful.

Even though it's like, in principle, it would be much easier to integrate AI's into the economy than it is to hire humans.

I think like the diffusion, I feel like, doesn't really explain.

I think diffusion is very real.

And doesn't have to, you know, doesn't exclusively have to do with limitation limitations on the AI models.

Like, again, there are people who use diffusion to, you know, as kind of a buzzword to say this isn't a big deal.

I'm not talking about that.

I'm not talking about, you know, AI will diffuse at the speed that previous.

I think AI will diffuse much faster than previous technologies have, but not infinitely fast.

So I'll, I'll just give an example of this, right?

Like, there's like, Claude code.

Like Claude code is extremely easy to set up.

You know, if you're a developer, you can kind of just start using Claude code.

There is no reason why a developer at a large enterprise should not be adopting Claude code as quickly as, you know, individual developer or developer at a startup.

And we do everything we can to promote it, right?

We sell, we sell Claude code to enterprises and big enterprises.

Like, you know, big, big financial companies, big pharmaceutical companies, all of them.

They're adopting Claude code much faster than enterprises typically adopt new technology, right?

But, but, but again, it, it, it, it, it takes time.

Like any given feature or any given product like Claude code or like co-work will get adopted by the, you know, the, you know, the individual developers who were on Twitter all the time by the like series A start-ups.

Many months faster than, you know, then they will get adopted by like, you know, like large enterprise that does food sales.

There are a number of factors like you have to go through legal.

You have to provision it for everyone.

It has to, you know, like it has to pass security and compliance.

The leaders of the company who are further away from the eye revolution, you know, are, are forward-looking, but they have to say, oh, it makes sense for us to spend 50 million.

This is what this Claude code thing is.

This is why it helps our company.

This is why it makes us more productive.

And then they have to explain to the people to levels below.

And they have to say, okay, we have 3,000 developers like, here's how we're going to roll it out to our developers.

And we have conversations like this every day like, you know, we are doing everything we can to make anthropics revenue grow 20 or 30x a year instead of 10x a year.

You know, and again, you know, many enterprises are just saying, this is so productive.

Like, you know, we're going to take shortcuts on our usual procurement process, right?

They're moving much faster than, you know, when we try to sell them just the ordinary API, which many of them use, but Claude code is a more compelling product.

But it's not an infinitely compelling product.

They don't think even aGI or powerful AI or country of geniuses in the data center will be an infinitely compelling product.

It will be a compelling product enough.

Maybe to get 3 or 5 or 10x a year growth even when you're in the hundreds of billions of dollars, which is extremely hard to do.

And it has never been done in history before, but not infinitely fast.

I buy that would be a slice though down.

And maybe this is not your claim.

But sometimes people talk about this like, oh, the capabilities are there, but because of diffusion.

Otherwise, like, we're basically at AGI and then I don't believe we're basically at AGI.

I think if you had the country of geniuses in a data center, if you were company to adopt the geniuses in a data center.

If you had the country of geniuses in a data center, we would know it.

You would know it.

If you had the country of geniuses in a data center.

Like, everyone in this room would know it.

Everyone in Washington would know it.

Like, you know, people in rural rural parts might not know it.

But like, but we would know it.

We don't have that now.

That's very clear.

As Daria was ending at, to get generalization, you need to train across a vibrating of realistic tasks and environments.

For example, with a sales agent, the hardest part is in teaching it to mash buttons in a specific database in sales force.

It's training the agent's judgment across ambiguous situations.

How do you sort through a database with thousands of leads to figure out which ones are hot?

How do you actually reach out?

What do you do when you get ghosted?

When an AI lab wanted to train a sales agent, Libobox brought in dozens of Fortune 500 sales people to build a bunch of different oral environments.

They created thousands of scenarios where the sales agent had to engage with the potential customer, which was roleplayed by a second AI.

Libobox made sure that this customer AI had a few different personas.

Because when you cold call, you have no idea who's going to be on the other end.

You need to be able to deal with a whole range of possibilities.

Libobox's sales experts monitor these conversations turned by turn, tweaking the roleplay agent to ensure that the kinds of things an actual customer would do.

Libobox could iterate faster than anybody else in the industry.

This is super important because oral is an empirical science.

It's not a soft problem.

Libobox has a bunch of tools for monitoring agent performance in real time.

This lets their experts keep coming up with tasks, so that the model stays in the right distribution of difficulty and gets the optimal rewards signal during training.

Libobox can do this sort of thing in almost every domain.

They've got hedge fund managers, radiologists, even airline pilots.

So whatever you're working on, Libobox can help.

Learn more at labelbox.com slash thorcash.

Coming back to concrete predictions because I think because there's so many different things to disseminate and be easy to talk past each other when we're talking about capability.

So for example, when I interviewed three years ago, I asked your prediction about what we should be expect three years from now.

I think you're right.

So when you said we should expect systems, which if you talk them for the course of an hour, it's hard to tell them apart from a generally well-educated human.

Yes.

And then you were right about that.

And I think spiritually I feel unsatisfied because my internal expectation was, was that such a system could automate large parts of white collar work.

And so it might be more productive to talk about the actual and capabilities you want such a system.

So I will basically tell you what, you know, where I think we are.

So let me ask you in a very specific question so that we can figure out exactly what kinds of capabilities we should be about to do.

So maybe I'll ask about it in the context of a job I understand well, not because it's the most relevant job, but just because I can evaluate the claims about it.

Take video editors, right?

I've video editors and part of their job involves learning about our audience's preferences, learning about my preferences and taste and the different trade-offs we have.

And just over the course of many months building up this understanding of context.

And so the skill and ability they have six months into the job, a model that can pick up that skill on the job on the fly.

When should we expect such an AI system?

Yeah.

So I guess what you're talking about is like, we're doing this interview for three hours.

And then like, you know, someone's going to come in, someone's going to edit it.

They're going to be like, oh, you know, I don't know.

Dario like, you know, scratched his head and, you know, we could edit that out.

And you know, that was like long, there was this like long discussion that like is less interesting to people.

And then, you know, then there's other thing that's like more interesting to people.

So, you know, let's let's let's kind of make this this edit.

So, you know, I think the country geniuses in a data center will be able to do that.

The way it will be able to do that is, you know, it will have general control of a computer screen, right?

Like it, you know, and you'll be able to feed this in.

And it'll be able to also use the computer screen to like go on the web, look at all your previous, look at all your previous interviews like look at what people are saying on Twitter and response to your interviews like talk to you, ask you questions, talk to your staff, look at the history of kind of edits and it's that you did.

And from that like, do the job.

Yeah.

So, I think that's dependent on several things.

One that's dependent.

And I think this is one of the things that's actually blocking deployment.

Getting to the point on computer use, where the models are really masters at using the computer, right?

And, you know, we've seen this climb in benchmarks and benchmarks are always, you know, in perfect measures.

But like, you know, OS world is, you know, went from, you know, like 5% of, you know, like I think when we first released, you know, computer use like a year and a quarter ago was like maybe 15% or I don't remember exactly, but we've climbed from that to like 65% or 70%.

And, you know, there may be harder measures as well, but, but I think computer use has to pass a point of reliability.

Can I just ask a follow up on that before you move in the next point?

I often, for years, I've been trying to build different internal LLM tools for myself.

And I often, I have these text and text out tasks, which should be dead center in the repertoire of these models.

And yet I still hire humans to do them, just because it's, if it's something like, make, I identify what the best clips would be in this transfer.

And maybe they'll do like a seven out of ten job at them.

But there's not this ongoing way I can engage with them to help them get better at the job, the way I could with a human employee.

And so that missing ability, even if you saw computer use, with still block my ability to like, offload an actual job to them.

Again, there's, there's, this gets back to what we're talking about before with learning on the job where it's, it's very interesting, you know, I think, I think with the coding agents like, I don't think people would say that learning on the job is what is, you know, preventing the coding agents from like, you know, doing everything end hand like they keep, they keep getting better.

We have engineers and anthropic who like don't write any code.

And when I look at the productivity to your previous question, you know, we have folks who say, this, this GPU kernel, this chip, I used to write it myself, I just have Claude do it.

And so there's this, there's this enormous improvement in productivity.

And I don't know, like when I see Claude like familiarity with the code base or like, you know, or, or a feeling that the model hasn't worked at the code, hasn't worked at the company for, for a year, that's not high up on the list of complaints I see.

And so I think what I'm saying is we're, we're like, we're kind of taking a different day out.

No, don't you think with coding, that's because there's an external scaffold of memory, which exist, instantiated in the code base, which I don't know how many other jobs have coding made fast progress precisely because it has its unique advantage that other economic activity doesn't.

But, but when you say that, what you're, what you're implying is that by reading the code base into the context, I have everything that the human needed to learn on the job.

So that would be an example of whether it's written or not, whether it's available or not, a case where everything you needed to know you got from the context window, right?

And that, and that, and that what we think of as learning, like, oh man, I started this job, it's going to take me six months to understand the code base, the model just did it in the context.

Yeah, I honestly don't know how to think about this because there are people who qualitatively report where you're saying, there was a meter study, I'm sure you saw last year.

Yes, where they had experienced developers who tried to close poor requests in repositories that they were familiar with.

And those developers reported an uplift, they reported that they felt more productive with the use of these models, but in fact, if you look at their output and how much was actually merged back in, there's a 20% downlift.

And so I'm trying to square the qualitative feeling that people feel with these models versus one in a macro level, where are all the, where is this like Renaissance of software, and two when people do these independent evaluations.

Why are we not seeing the, yeah, so creative benefit to be able to expect.

Within anthropic, this is just really unambiguous, right?

We're under an incredible amount of commercial pressure and making even harder for ourselves, because we have all this safety stuff we do that I think we do more than than than other companies.

So like the the the pressure to survive economically, while also keeping our values is is just incredible, right?

We're trying to keep this 10x revenue curve going there's like there is zero time for bullshit.

There is zero time for feeling like we're productive when we're not like these tools make us a lot more productive like why why do you think we're concerned about competitors using the tools because we think we're ahead of the competitors.

And like we don't we don't want to sell we we wouldn't be going through all this trouble if this was secretly reducing or reducing our productivity like we see the end productivity every few months in the form of model launches like there's no kidding yourself about this like the models make you more productive.

And that people feeling like they're productive is qualitatively predicted by studies like this, but to if I just look at the end output obviously you guys are making fast progress, but the fact you know the idea was supposed to be with recursive some improvement is that you make a better AI the AI helps people to better next AI etc.

And when I see instead if I look at the you open AI deep mind is that people are just shifting around the podium every few months.

Maybe you think that stops because you're one or whatever, but but why are we not seeing the person with the best coding model have this lasting advantage if in fact there are these enormous productivity gains from the last.

So no no no I mean I mean I mean I think it's all like my my model of the situation is there's there's an advantage that's gradually growing like I would say right now.

The coding models give maybe I don't know a like 15 maybe 20% total factor speed up like that's my view and six months ago it was maybe 5% and so and so it didn't matter like 5% doesn't register.

And now just getting to the point where it's like one of several factors that that kind of matters and and that's going to that's going to keep speeding up and so I think six months ago like you know.

But there were several there were several companies that were at roughly the same point because you know this this wasn't this wasn't an notable factor but I think it started to speed up more and more.

I you know I would I would also say they're multiple companies that you know right models that are used for code and you know we're not perfectly good at you know preventing some of these other companies from from from from from from kind of using our models internally.

So you know I think I think everything we're kind of kind of everything we're seeing is consistent with this kind of this kind of snowball model where you know there's no hard again my my my my my my theme and all of this is like all of this is soft take off like soft smooth exponentials although the exponentials are relatively steep.

And so and so we're seeing this snowball gather momentum where it's like 10% 20% 25% you know for for 40% and as you go yeah and those all you have to get all the like things that are preventing you from from closing the loop out of the way but like this is one of the biggest priorities within and traffic.

As stepping back I think before in the stack we were talking about.

Well when do we get this on the job learning and it seems like the coding the point you're making the coding thing is we actually don't need on the job learning that you can have tremendous productivity improvements you can have.

But potentially trillions of dollars a revenue for eye companies without this basic human ability maybe that's not going to make sure clarify.

But without this basic human ability to learn on the job but I just will go and like in in most domains of economic activity people say I hired somebody they were in that useful for the first few months and then over time.

They built up the context understanding it's actually harder to find what we're talking about here but they got something and then now now they're they're power horse and they're so valuable to us.

And if AI doesn't develop this ability to learn on the fly I'm not I'm a bit skeptical that we're going to see.

Huge changes to the world.

So I think I think I think two things here right there's the state of the technology right now which is again we have these two stages we have the pre training RL stage where you throw you throw a bunch of data and tasks into the models and then they generalize so it's like learning but it's like learning from more data and not.

You know not learning over kind of one human or one models lifetime so again this is situated between evolution and and and human learn but once you learn all those skills you have them and just like with pre training just how the models.

No more you know if if I look at a pre train model you know it knows more about the history of samurai in Japan than I do it knows more about baseball than I do it knows you know it knows more about you know.

Low pass filters and electronics then you know all all of these things it's knowledge is way broader than mine so I think I think even even just that.

You know may get us to the point where the models are better at you know kind of better at everything and then we also have again just with scaling the kind of existing setup we have the in context learning which I would describe as kind of like human on the job learning but like a little weaker and a little short term like you look at.

In context learning the you give the model a bunch of examples it does get it there's real learning the happens in context and like a million tokens is a lot that's that's you know that can be days if you in learning right you know if you think about the model you know.

You know kind of reading reading a million words you know it you know takes me how long would it take me to read a million I mean you know like days or weeks at least.

So you have these two things and and I think these two these two things within the existing paradigm may just be enough to get you the country's new assistant data center.

I don't know for sure but I think they're going to get you a large fraction of it there may be gaps but I certainly think just as things are this I believe is enough to generate trillions of dollars of revenue that's one that's all one.

I think that's one of the two is this idea of continual learning this idea of a single model learning on the job.

I think we're working on that too and I think there's a good chance that in the next year or two we also make we also solve that.

I you know I think you get most of the way there without it I think the trillions of dollars of you know that I think the trillions of dollars a year market.

Maybe all the national security implications and the safety implications that I wrote about in that a lesson of technology can happen without it but I I also think we and I imagine others are working on it.

And I think there's a good chance that that you know that we get there within the next year or two there are a bunch of ideas I won't go into all of them in detail but.

You know one is just make the context longer there's there's nothing preventing longer context from working you just have to train at longer context and then learn to to serve them at inference and both of those are engineering problems that we are working on and that I would assume others are working on as well.

So this context land increase it seemed like there was a period from 2020 to 2023 where from GPD 3 to GPD 4 to wherever there was an increase from like 2000 context lying still 128k.

If you're like for the next for the two-ish year since then we've been in the sameish ballpark yeah and when model context lines get much longer than that people report qualitative degradation in the ability to the model to consider that full context.

So I'm curious what you're internally saying that makes you think like oh 10 million context 100 million context to get human like six months learning building building context isn't a research problem this is a this is an engineering and inference problem right if you want to serve long context you have to like.

Story your entire KV cash you have to you know you know it's it's it's difficult to store all the memory in the GPUs to juggle the memory around I don't even know the detail you know at this point this is at a level of detail that that that that I'm no longer able to follow although you know I knew it in the GP D3 era of like you know these are the weights these are the activations you have to store.

But you know you know these days the whole thing is flip because we have MOE models and and and kind of all that but.

And and this degradation you're talking about like again without getting too specific like a question I would ask is like.

There's two things there's the context length you train at and there's a context length that you serve at.

If you train at a small context length and then try to serve at a long contracts length like maybe you get these declarations.

It's better than nothing you might still offer it but you get these declarations and maybe it's harder to train at a long context length yeah I so you know there's there's a lot.

I want to at the same time ask about like maybe some rabbit holes of like well wouldn't you expect that if you have to train on longer context length that would mean that.

You're able to get sort of like less samples in for the same amount of compute before maybe it's not worth diving deep on that I want to get an answer to the bigger picture question which is like okay so.

I don't feel a preference for a human editor that's been working for me for six months versus any I that's been working with me for six months.

What year do you predict that that will be the case.

I my I mean you know my guess for that is you know that there's there's a lot of problems that are basically like we can do this when we have the country of geniuses in the data center.

And so you know my my my my my my picture for that is you know again if you if you if you if you know if you made me guess it's like one to two years maybe one to three years it's really hard to tell I have a I have a strong view.

And then I have a hunch this is more like a 50 50 thing that it's going to be more like one to two maybe more like one to three so one to three years.

The country of geniuses.

And then the slightly less economically valuable task of editing videos.

And it seems pretty economically valuable let me tell you it's just there are a lot of use cases like that right there are a lot of similar.

So you're predicting that within one to three years.

And in generally anthropic is predicted that by late 26 or 27 we will have a systems that are quote.

Have the ability to navigate interfaces available to humans doing digital work today.

Intellectual capabilities mashing or exceeding that of noble price winners.

And the ability to interface with the physical world.

And then you give an interview two months ago with dealbook where you're emphasizing your.

Your company's more responsible compute scaling as compared to your competitors.

And I'm trying to square these two views where if you really believe that we're going to have a country of geniuses you.

You want as big a data center as you can get there's no reason to slow down the tam of a noble price winner.

And actually can do everything in all price winner can do is like trillions of dollars.

And so I'm trying to square this conservatism.

Which seems rational if you have more moderate timelines with your stated views about AI progress.

Yeah, so so it actually all fits together and we go back to this fast but not infinitely fast diffusion.

So like let's say that we're making progress at this rate.

You know the the technology is making progress this fast again.

I have you know very high conviction that like it's going you know the you know we're we're going to get there within within a few years.

I have a hunch that we're going to get there within a year or two.

So a little uncertainty on the technical side but like you know pretty pretty strong confidence that it won't be off by much.

What I'm less certain about is again the economic diffusion site like I really do believe that we could have models that are a country of geniuses.

A hundred country of geniuses in the data center in one to two years.

One question is how many years after that do the trillions and you know do the trillions and revenue start rolling in.

I don't think it's guaranteed that it's going to be immediate.

You know I think it could be one year it could be two years.

I could even stretch it to five years although I'm like I'm skeptical of that.

And so we have this uncertainty which is even if the technology goes as fast as I suspect that it will.

We don't know exactly how fast it's going to drive revenue.

We we know it's coming but with the way you buy these data centers if you're off by a couple years that can be ruinous.

It is just like how I wrote you know in machines of loving grace.

I said look I think we might get this powerful AI this country using the data center that description you gave comes from the machines of loving grace.

I said we'll get that twenty twenty six maybe twenty seven again.

That is that is my hunch wouldn't be surprised if I'm off by a year or two but like that is my hunch.

Let's say that happens that's the starting gun.

How long does it take to cure all the disease right that's that's one of the ways that like drives a huge amount of of of of economic value right like.

You cure you cure every disease you know there's a question of how much of that goes to the pharmaceutical company to the AI company.

But there's an enormous consumer surplus because everyone you know everyone you know everyone can get access for everyone which I care about greatly.

We you know we cure all of these diseases.

How long does it take you have to do the biological discovery you have to you know go you have to you know.

Man you've actually the new drug you have to you know go through the regulatory problem.

We saw this with like vaccines and COVID right like it that there's just this we got the vaccine out to everyone but it took a year and half right and and so my question is.

How long does it take to get the cure for everything which AI is the genius that can in theory and that out to everyone.

How long from when that AI first exists in the lab to when diseases have actually been cured for everyone right.

And you know we've had a polio vaccine for 50 years we're still trying to eradicate it in the most remote corners of Africa and you know the Gates Foundation is trying as hard as they can others are trying as hard as they can.

But you know that's difficult again I you know I don't expect most of the economic diffusion to be as difficult as that right that's like the most difficult case.

But but there's a there's a real dilemma here and and where I've settled on it is it will be it will be it will be faster than anything we've seen in the world but it still has its limits and and so then when we go to buying data centers.

And again again the curve I'm looking at is okay we you know we've had a 10x a year increase every year so beginning of this year we're looking at 10 billion in in annual and you know rate of annualized revenue with the being in the year we have to decide how much compute to buy.

And you know it takes a year or two to actually build out the data centers to reserve the data center so basically I'm saying like in twenty twenty seven how much compute do I get well I could assume.

That the revenue will continue growing 10x a year so it will be one 100 billion at the end of twenty twenty six and one trillion at the end of twenty twenty seven.

And so I could buy a trillion dollars actually would be like five trillion dollars of compute because it would be a trillion dollar a year for for five years right I could buy a trillion dollars of compute.

That starts at the end of twenty twenty seven and if my if my revenue is not a trillion dollars if it's even.

800 billion there's no force on earth there's there's no hedge on earth that could stop me from going bankrupt if I if I buy that much compute and so even though a part of my brain wonders if it's going to keep going 10x.

I can't buy a trillion dollars a year of compute in in in in in in in in twenty twenty seven.

If I'm just off by a year in that rate of growth or if the growth rate is five x a year instead of 10x a year then then you know the you go bankrupt.

And so you end up in a world where you know you're supporting hundreds of billions not trillions and you accept you accept some risk that.

There's so much demand that you can't support the revenue and you accept still some risk that you know you got it wrong and it still and so when I talked about behaving responsibly.

What I meant actually was not the absolute amount that that actually was not you know I think it is true we're spending somewhat less than some of the other players it's actually the other things like.

Have we been thoughtful about it or are we yolo even saying oh we're going to do a hundred billion dollars here a hundred billion dollars there.

I kind of get the impression that you know some of the other companies have not written down the spreadsheet that they don't really understand the risk they're taking they're just kind of doing stuff because it sounds cool.

And and we thought carefully about it right we're an enterprise business therefore you know we can rely more on revenue it's less fickle than consumer we have better margins which is the buffer between.

And so I think we bought an amount that allows us to capture pretty strong upside worlds it won't capture the full 10x a year and things would have to go pretty badly for us to be for us to be in financial trouble so I think we thought carefully and we've made that balance and and that's what I mean when I say that we're being responsible.

It seems like it's possible that we actually just have different definitions as a country of a genius in a data center because when I think of like actual human genius is an actual country of human genius is a data center I'm like.

I believe by five trillion dollars over the computer to run actual culture of human genius is a data center so let's say JP Morgan or Moderna or whatever it doesn't want to use them.

I've got a country of geniuses let's build those start their own company and if like they can answer their own company and their bottleneck by clinical trials it is worth stating with clinical trials like most clinical trials failed because the drug doesn't work there's not efficacy right.

I make exactly that point in machines of love and grace I say the clinical trials are going to go much faster than we're used to but not not instantly not infinitely fast.

And then suppose it takes a year to for the clinical trials to work out so that you're getting revenue from that and can make more drugs.

Okay, well you've got a country of geniuses and you already I lab and you have you could use many more every searchers and you also think there's these like self reinforcing gains from you know smart people working on AI tech so like okay you can have that right but you can have the data center working on like AI progress there more gains from buying.

Like substantially more gains from buying a trillion dollars a year of compute versus 300 billion dollars a year of compute if your competitor is buying a trillion yes there is.

Well then no there's some gain but then but again that there's this chance that they go bankrupt before.

You know be again if you're off by only a year you destroy yourselves that's that that's the balance we're buying a lot we're buying a hell of a lot like we're not we're you know we're buying an amount that's.

Comparable to that that you know that the biggest players in the game are buying but but if you're asking me why why haven't we signed.

You know 10 10 trillion of compute starting and starting in mid twenty twenty seven first of all can't be produced there isn't that much in the world.

But but second what is the country of geniuses comes but it comes in mid twenty twenty eight instead of mid twenty twenty seven you go bankrupt so.

If your projection is one to three years it seems like you should want 10 trillion dollar compute by.

Twenty twenty nine money twenty and maybe twenty twenty twenty eight is like I mean you know you think are you're interested like it seems like.

Even in your the longest version of the timelines you stayed the compute you are wrapping up to build doesn't seem.

What what what makes you think that.

Well you as you said you want the 10 trillion like human wages let's say are.

On the order of 50 trillion a year if you look at so so I won't I won't talk about anthropic in particular but if you talk about the industry like.

The amount of compute the industry hit you know the the amount of compute the industries building.

This year is probably in the you know I don't know very low tens of you know call it 10 15 gigawatts next year I you know it goes up by roughly three acts a year so like next year's third year 40 gigawatts and twenty twenty eight might be a hundred twenty twenty nine might be like.

Three three hundred gigawatts and like each gigawatt costs like.

Um maybe ten I mean I'm doing the math in my head but each gigawatt costs maybe ten billion dollar you know or border ten to fifteen billion dollars a year so.

You know you kind of you you know you put that all together and you're getting about about what you describe you're getting multiple trillions a year by twenty twenty eight twenty twenty nine so you're you're getting exactly that.

You're getting you're getting exactly what you predict.

Um that's for the industry that that's for the industry.

That's right.

Suppose that anthropic compute keeps three acts a year and then by like twenty seven you have.

Or twenty seven twenty eight you have ten gigawatts and like multiply that by as you say.

Ten billion so then it's like a hundred billion a year but then you're saying the term by twenty twenty eight nine.

I don't want to give exact numbers for anthropic but but these numbers are too small these numbers are too small.

Okay interesting.

I'm really proud that the puzzles I've worked on with Jane Street have resulted in them hiring a bunch of people for my audience.

Well they're still hiring and they just send me another problem.

For this one and they spent about twenty thousand GPU hours training back doors into three different language models.

Each one has a hidden prompt that elicits completely different behavior you just have to find the trigger.

This is particularly cool because finding back doors is actually an open question in front of your AI research.

And anthropic actually released a couple of papers about celebrations and they show that you can build a simple classifier on the residual stream to detect when a back door is about to fire.

But they already knew what the triggers were because they built them year you don't and it's not feasible to check the activations for all possible trigger phrases.

Unlike the other puzzles they made for this podcast.

Jane Street isn't even sure this one is solvable but they've set aside fifty thousand dollars for the best attempts and write-ups.

And the puzzles live at Jane Street dot com slash work ash and they're accepting submissions until April 1st.

All right back to Daria.

You've told investors that you plan to be profitable starting in 28 and this is the year where we're like potentially getting the country of geniuses at data center.

And this is like going to now unlock all this progress and medicine and health and et cetera et cetera and new technologies.

Wouldn't this be a particular the exactly the time where you'd like want to reinvest in the business and build bigger countries.

I mean profit of profitability is this kind of like weird thing in this field.

Like I don't think I don't think in this field profitability is actually a measure of you know kind of spending down versus investing in the business.

Let's let's just let's just take a model this I actually think profitability happens when you underestimate the amount of demand you were going to get and loss happens when you over estimated the amount of demand you were going to get.

Because you're buying the data centers ahead of time so think about it this way.

Ideally you would like and again these are stylized facts these numbers are not exact for and I'm just trying to make a toy model here.

Let's say half of your compute is for training and half of your compute is for inference.

And you know the inference has some gross margin that's like more than 50%.

And so what that means is that if you were in steady state you build a data center if you knew exactly exactly exactly the demand you were getting you would.

You would you would you would get a certain amount of revenue say I don't know let's say you pay a hundred billion dollars a year for compute.

And on fifty billion dollars a year you support a hundred fifty billion dollars on of of of of revenue and the other fifty billion the other fifty billion are used for training.

So basically your profitable you make you make you make fifty billion dollars a profit those are the economics of the industry today or sorry not today but like that's where we're.

Where we're projecting forward in a year or two the only thing that makes that not the case is if you get.

Less demand than fifty billion then you have more than fifty percent of your your data center for research and you're not profitable so you you know you train stronger models but you're like not profitable.

If you get more demand than you thought then your research gets squeezed but you know you're you're kind of able to support more inference and you're more profitable so it's.

Maybe I'm not explaining it well but but the thing I'm trying to say is you decide the amount of compute first and then.

You have some target desire of inference versus.

Versus training but that gets determined by demand it doesn't get determined by what I'm hearing is the reason you're predicting profit is that you are systematically underestimate under investing in compute right because if you actually I'm saying I'm saying it's hard to predict so so these things about.

2028 and what it will happen that's our that's our attempt to do the best we can with investors all of this stuff is really uncertain because of the cone of uncertainty like.

We could be profitable in twenty twenty six if the if the revenue grows fast enough and then and then.

You know if we if we overestimate or underestimate the next year that could swing wildly like I I what I'm trying to get is you have a modeling your head of like.

The the business invest invest invest invest get scale and and and and kind of then becomes profitable there's a single point at which things turn around.

I don't think the economics of this industry work that way.

I see so if I'm understanding correctly you're saying because of the discrepancy between the amount of compute we should have gotten and the amount of compute we got we.

We were like sort of forced to make profit but that doesn't mean we're going to continue making profit we're going to like.

Reinvest the money because well now he has made so much progress and we want the bigger country of geniuses and so.

And then back into revenue is high but losses are also high if we if we if we predict if every year we predict exactly what the demand is going to be will be profitable every year.

Because grow because spending spending 50% of your compute on on 50% of your compute on research roughly.

And plus a gross margin that's higher than 50% and correct demand prediction leads to profit that's the profit that's that's the profitable business model that I think is kind of like.

There but like obscured by these like building ahead in prediction errors I guess you're treating the 50% as a.

As a sort of like you know just like a given constant where as you in fact if you if the I progress is fast and you can increase the progress by scaling up more you just have more the 50% and not make progress what I'll say you might want to scale up it more you might want to scale it up more but but but you know remember the log returns to scale right.

If 70% would get you a very little bit of a smaller model through a factor of of 1.4x right like that extra 20 billion dollars is is is is you know that each each dollar there is worth much less to you because it because because the log linear set up.

And so you might find that it's better to invest that that that that it's better to invest that 20 billion dollars in you know in serving inference or in hiring engineers who are who are kind of better who are kind of better who are kind of better what they're doing.

So the the reunited said 50% that's not that's not exactly our target it's not exactly going to be 50% it'll probably vary vary over time or what I'm saying is the the the like log linear return what it leads to is you spend of order one.

And it's not a fraction of the business right like not 5% not 95% and then it then it that you know then then that you get diminishing returns because of the because of the wall.

And we're talking about diminishing returns and after you're spending 50 billion a year right like this is a point I'm sure you'd make but like diminishing returns on a genius is could be quite high and more generally like what is profit in the market economy profit is basically saying.

The other companies in the market can like do more things with this money that I can put aside and drop I'm just trying to like because I you know I don't want to give information about anthropic is why I'm giving these stylized numbers but like let's just derive the equilibrium of the industry right I think the so so so why doesn't everyone spend 100% of their.

You know 100% of their compute on training and not serve any customers right it's because if they didn't get any revenue they couldn't raise money they couldn't do compute deals they couldn't buy more compute the next year so there's going to be an equilibrium where every.

Every company spends less than 100% on on on on on on training and certainly less than 100% on inference it should be clear why you don't just serve the current models and and you know and and and never train another model because then.

You don't have any demand because you'll because you'll fall behind so there's some equilibrium it's it's not going to be 10% it's not going to be 90% let's just say as a stylized fact it's 50% that's what I'm getting that and and I think we're going to be in a position where.

That equilibrium of how much you spend on training is less than the gross margins that that you're that you're able to get on compute and so the the underlying economics are profitable the problem is you have this this hellish demand prediction problem when you're when you're buying the next year of compute and you might guess under and.

The very profitable but have no compute for research or you might guess over and you know you're you're you're.

You are not profitable and you have all the compute it could compute for research and work.

Does that make sense just as a dynamic model of industry maybe stepping back I'm like.

I'm not saying I think the country of geniuses is going to come in two years and therefore you should buy this compute.

To me what you're seeing the end conclusion you're arriving at makes a lot of sense but that's because like oh it seems like country geniuses is hard and there's a long way to go and so the stepping back the thing I'm trying to get it is more like.

It seems like your world is compatible with somebody who says we're like 10 years away from a world in which like we're generating trillion dollars just that's just not my view yeah that is that is not my view like.

I so so all like I'll like make another prediction it is hard for me to see that that there won't be trillions of dollars in revenue before 2030.

Like I can construct a plausible world it takes maybe three years so now that would you know that would be the end of what I think it's plausible like in 2028.

We get the real country of geniuses in the data center you know the revenues been been go you know the revenues been going into the maybe is it is in the low hundreds of billions by by by by 2028.

And and and then the country of geniuses accelerates it to trillions you know and and basically on the slow end of diffusion it takes two years to get to the trillions.

That that would that that that that that that that will be the world where it takes until that will be the world where it takes until 2030.

I I suspect even composing the technical exponential and diffusion exponential will get there before 2030.

So you laid out a model where anthropic makes profit because it seems like fundamentally we're in a compute constrained world and so it's like.

Eventually we keep growing computer.

No I think I think the way the profit comes is again and and you know let's let's just abstract the whole industry here like we have a.

We have a you know let's just imagine we're we're in like an economics textbook we have a small number of firms each can invest a limited amount.

And you know or like each can invest some fraction fraction and R&D they have some marginal cost to serve the margins on that the profit margin the gross profit margins on that marginal cost are like very high because because because inference is efficient.

There's some competition but the models are also differentiated there's some there's some you know companies will compete to push their research budgets up but like because there's a small number of players you know we have the.

What is it called an economic or no equilibrium I think is what the what the small number of firm equal equilibrium is it the point is it doesn't equilibrate to perfect competition with with with with with with with zero margins if there's like.

Three firms if there's three firms in the economy all our kind of independently behaving behaving rationally it doesn't equilibrate to zero.

Help me understand that because right now we do have three leading firms and they're not making profit.

And so what yeah what what is changing.

Yeah so the again the gross margins right now are very positive what's happened what what's happening is a combination of two things one is we're still in the exponential scale up phase of compute.

So what basically what that means is we're training like a model gets trained yeah it costs you know let's say a model got trained that costs a billion dollars last year.

And then this year it produced four billion dollars of revenue and cost one billion dollars to to to to to to inference from so you know again I'm using stylized number here but you know the 75% you know gross gross gross margins.

And you know this this 25% tax so that model as a whole.

Makes two billion dollars.

But at the same time we're spending ten billion dollars to train the next model because there's an exponential scale up and so the company loses money each model makes money.

But the company loses money the equilibrium I'm talking about is an equilibrium where we have the country of geniuses we have the country of geniuses in the data center.

But that that model training scale up has a quillibrated more maybe it's still it's still going up we're still trying to predict the demand but it's more it's more leveled out.

I'll give you a couple things there so let's start with the current world in the current world you're right that as you said before if you treat each individual model as a company it's profitable.

Of course a big part of the production function of being a frontier lab is training the next model right so that if you didn't do that then you'd make profit for two months.

That's right and you wouldn't have margins because you wouldn't have the best model and then so yeah you can be profits too much in terms of some point that reaches the biggest scale that it can reach.

And then and then in equilibrium we have algorithmic improvements but we're spending roughly the same amount to train the next model as as we as we spent to train the current model.

So this equilibrium relies I mean at some point it's at some point you run out of money in the economy.

The fixed length of labor follows the economy is going to grow right that's not your predictions.

Well we're going to have this but this is another example of the theme I was talking about which is that the economy will grow.

Much faster with AI then I think it ever has before but it's not right right now the computer's growing 3x a year.

Yeah I don't believe the economy is going to grow 300% a year.

Like I said this in machines of loving grace like I think we may get 10 or 20% per year growth in the economy but we're not going to get 300% growth in the economy.

So I think in the end you know if compute becomes the majority of what the economy produces it's going to be kept by that.

So that's okay now let's assume a model where compute stays capped.

The world where frontier labs are making money is one where they continue to make fast progress because fundamentally your margin is limited by.

How good the alternative is and so you are able to make money because you have a frontier model if you did not hurt your model you wouldn't be making money.

And so this model requires.

They're never to be a steady state like forever and ever you actually making.

No I don't think that's true.

I mean I feel like we're like we're taught we're you know we're the feel like this is an economics.

Like you know this is like an economics.

We never stop talking about economics so no but but there there are worlds in which.

So I don't think this field's going to be a I don't think this field's going to be a monopoly on my lawyers never want me to say it or not.

But I don't think this field's going to be a monopoly but but you do get.

You get industries in which there are small number of players not one but a small number of players and.

Or narrowly like the way you get monopolies like.

I think that's the way you get industries in which there are small number of players are very high cost of entry right.

So you know a cloud is like this I think cloud is a good example of this you have three maybe four players within cloud I think I think that's the same.

And the reason is that it's it's so expensive it requires so much expertise and so much capital to like run a cloud company right and so you have to put up all this capital.

And then in addition to putting up all this capital you have to get all this other stuff that like you know requires a lot of skill to you know to make it happen and so it's like if you go to someone and you're like I want to disrupt this industry.

I think that's 100 billion dollars or like okay I'm putting a hundred billion dollars and also betting that you can do all these other things that these people have been doing.

You create the profit in the industry and then and then the effect of your entry is this is the profit margins go down so you know we have equilibrium like this all the time in the economy where we have a few we have a few players.

The profits are not astronomical margins are not astronomical but they're they're not zero right and and you know I think I think that's what we see on cloud cloud is very undifferentiated models are more differentiated than cloud right like everyone knows Claude is Claude is good at different things then GPT is good at is then then Gemini is good at and it's not just Claude is good at coding GPT is good at you know math and reasoning you know.

It's more subtle than that like models are good at different types of coding models have different styles like I think I think these things are actually you know quite different from each other and so I would expect more differentiation than you see in cloud now there there actually is a.

There is one counter argument and that counter argument is that if all of that the process of producing models becomes.

If AI models can do that themselves then that could spread throughout the economy but that is not an argument for commoditizing AI models in general that's kind of an argument for commoditizing the whole economy at once.

I don't know what what quite happens in that world where basically anyone can do anything anyone can build anything and there's like no mode around anything at all.

I mean I don't know maybe we want that world like like maybe that's the maybe that's the end state here like maybe maybe you know when maybe when when when kind of AI models can do you know when when when when AI models can do everything if we've solved all the safety and security problems like you know that's one of the one of the mechanisms for you know.

You know, you know just just kind of the economy flatten itself again but but that's kind of like post like far post countries.

This isn't a data center.

Maybe a finer way to put that potential point is one.

It seems like AI research is especially loaded on raw intellectual power which will be especially a button and then a world with a GI.

If you just look at the world today there's very few technologies that seem to be diffusing as fast as.

As AI algorithmic progress and so the does hint that this industry sort of structurally diffusive.

So I think coding is going fast but I think AI research is a super set of coding and their aspects of it that are not going fast.

But I do think again once we get coding once we get AI models going fast then you know AI you know that will speed up the ability of AI models to kind of do everything else.

So I think while coding is going fast now I think once the AI models are building the next AI models and building everything else.

The kind of whole the whole economy will side it kind of go at the same pace.

I am I am worried geographically though.

I'm a little worried that like just proximity to AI having heard about AI that that may be one differentiator.

And so when I said the like you know 10 or 20 percent growth rate.

A worry I have is that the growth rate could be like 50 percent in Silicon Valley and you know parts of the world that are kind of socially connected to Silicon Valley.

And you know not that much faster than its current pace elsewhere and I think that be a pretty messed up world so one of the things I think about a lot is how to prevent that.

Yeah.

Do you think that once we have this computer genius is a data center that robotics is sort of quickly solved afterwards because it seems like a big problem with the robotics is that.

A human can learn how to tell operate current hardware but current AI models can at least not not in a way that's super productive.

And so if we have this ability to learn like a human should it solve robotics immediately as well.

I don't think it's dependent on learning like a human it could happen in different ways again.

We could have trained the model on many different video games which are like robotic controls or many different simulated robotics environments or just you know train them to control computer screens and they learn to generalize so it will happen.

It's not necessarily dependent on human like learning human like learning is one way it could happen if the models like oh I pick up a robot I don't know how to use it I learned.

That could happen because we discovered a discovering continual learning that could also happen because we train the model on a bunch of environments and then generalized or it could happen because the model learns that in the context like.

It doesn't actually matter which way if we go back to discussion we had like like an hour ago that type of thing can happen in that type of thing can happen in several different ways.

But I do think when for whatever reason the models have those skills then robotics will be revolutionized both the design of robots because the models will be much better than humans at that.

And also the the ability to kind of control robots so we'll get better at the physical building the physical hardware building the physical robots and we'll also get better at controlling it now you know does that mean the robotics.

Industry will so be generating trillions of dollars of revenue my answer there is yes, but there will be the same extremely fast but not infinitely fast diffusion so will robotics be be revolutionized yeah maybe tack on another year or two.

That's that's that's that's the way I think about these things.

There's a general skepticism about extremely fast progress.

The here here's maybe which is like it sounds like you are going to solve continual learning what we're in other within the matter of years.

But just as people weren't talking about continual learning a couple of years ago and then we realized oh why are these models as useful as they could be right now even though they are clearly passing the touring test and are experts in so many different domains maybe it's this thing.

And then we saw this thing and we realized actually there's another.

Another thing that human intelligence can do and that's a basis of human labor that these models can't do and then why not think there will be more things like this I think that like.

Where you know we've found the pieces of human intelligence well to be clear I mean I think continual learning as I said before might not be a barrier at all right like like you know I think I think we may be just get there by.

I mean generalization and and and and and our L generalization like I think there but just might not be.

There basically might not be such a thing at all in fact I would point to the history in in ML of.

People coming up with things that are barriers that end up kind of dissolving within the big blob of compute right that you know people talked about you know.

How do you have.

You know how do how do your models keep track of nouns and verbs and you know how do they you know they can understand cement syntactically but they can't understand semantically you know.

It's only statistical correlations you can understand a paragraph you can understand a word there's reasoning you can't do reasoning but then suddenly it turns out you can do code and math very well at all so.

I think there's there's actually a stronger history of some of these things seeming like a big deal and then and then kind of and then kind of dissolving some of them are real I mean the need for data's real maybe continual continual learning is a real thing but again.

I would ground us in something like code like I think we may get to the point in like a year or two where the models can just do sweet and tend like that's a whole task that's a whole sphere of human activity that that we're just saying models can do it now.

When you say end to end do mean.

Setting technical direction understanding the context of the problem yes yes I mean all of that interesting I mean.

That that is if you're like AJ I complete.

There's maybe is internally consistent but it's not like saying 90% of code or 100% of code it's like no no no no no I gave this I gave this spectrum 90% of code 100% of code 90% of end tense we 100% of end tense we new tasks are created for trees eventually those get done as well but the long Spanish on there but we're traversing the spectrum very quickly yeah.

I think it's funny that I've seen a couple of podcasts you've done where the host will be like I would work after the session by the control learning thing and it always makes you crack off because you're like you know you've written a I researcher for like 10 years.

I'm sure there's like some feeling of like okay so podcasts are what and that's it.

You know the truth the truth of the matter is that we're all trying to figure this out together right there there are some ways in which.

I'm able to see things that others aren't these days that probably has more to do with like I can see a bunch of stuff within and throw up and after make a bunch of decisions then I have any great research insight that that others don't right I you know.

I mean in 2500 person company like it's it's actually pretty hard for me to have a concrete research insight you know much harder than you know then then what have been you know 10 years ago or you know we're even two or three years ago.

As we go towards a world of a full drop in remote work or replacement does a API pricing model still make the most sense and if not what is a correct way to price API or survey GI.

Yeah, I mean I think there's going to be a bunch of different business models here sort of all at once that are going to be that are going to be experimented with.

I actually do think that the the API model is is more durable than many people think one way I think about it is if the.

Technology is kind of advancing quickly if it's advancing exponentially what that means is there's there's always kind of like a surface area of kind of new use cases that have been developed in in the last in the last three months.

Any kind of product surface you put in place is always at risk of sort of becoming irrelevant right any given product surface probably makes sense for our you know a range of capabilities of the model right the the chatbot is already running into limitations of you know making it smarter doesn't really help the average consumer that much.

But I don't think that's a limitation of AI models I don't think that's evidence that you know the models are are the models are good enough and they're there you know them getting better doesn't matter to the economy doesn't matter to that particular product.

And and so I think the value of the API is the API always offers an opportunity you know very close to the bare metal to build on what the latest thing is and so there you know there's there's there's kind of always going to be this you know this this kind of.

Front of new startups and new ideas that weren't possible a few months ago and are possible because the model is advancing and and so I actually I I kind of actually predict that we are.

It's going to exist alongside other models but we're always going to have the API business model because there's there's always going to be a need for a thousand different people to try experimenting with the model in different way.

100 of them become startups and 10 of them become big successful startups and you know two or three really end up being the the way that people use the model of a of a given generation so I basically think it's always going to exist at the same time.

Sure there's going to be other models as well like not every token that's output by the model is worth the same amount think about you know how what is the value of the tokens that are like you know that the model outputs when someone you know.

Someone you know calls them up and says my mac isn't working or something you know the models like restart it right and like you know someone hasn't heard that before but like you know the model said that like.

10 million times right you know that that maybe that's worth like a dollar or a few cents or something.

Whereas if the model you know the model goes to you know one of the one of the pharmaceutical companies and it says you know this molecule you're developing you should take the aromatic ring from that end of the molecule and put it on that end of molecule.

And and you know if you do that wonderful things will happen like like those tokens could be worth you know that's a millions of dollars right.

So so I think we're definitely going to see business models that that recognize that you know at some point we're going to see you know pay for results or you know in some in some form or we may see forms of compensation that are like labor.

You know that that kind of work by the hour I you know I don't know I think I think because it's new industry a lot of things are going to be tried and I you know I don't know what will turn out to be the right thing.

What I find I take your point that people will have to try things to figure out what is the best way to use this blob of intelligence but what I find striking is.

So I don't think in the history of startups there has been a single application that has been as hotly competed in has coding agents and.

And and the cloud code is a category leader here and that seems surprising to me like it doesn't seem intrinsically like anthropic hat to build this and I wonder if you have an accounting of why it had to be anthropic or why how anthropic ended up building an application in addition to the model underlying it.

So it actually happened in a pretty simple way which is we had our own.

You know we had our coding models which were good at coding and and you know around the beginning of 2025 I said I think the time is come where you can have non trivial acceleration of your own research.

If you're an AI company by using these models and of course you know we you need an interface need a harness to use them and so I encourage people internally and I didn't say this is one thing that you know that you have to use I just said people should experiment with this.

And then you know this thing I think it might have been originally called called CLI and then the name eventually got changed to cloud code internally.

Was the thing that kind of everyone was using and it was seen fast internal adoption and I looked at it and I said probably we should launch this externally right.

You know it's it's seen such fast adoption within anthropic like you know like you know coding is a lot of what we do and so you know we have a we have an audience of many many hundreds of people that's in some ways at least representative of the external audience so.

It looks like we already have product market fit let's launch this thing and and then we launched it and and I think you know just just the fact that.

We ourselves are kind of developing the model and we ourselves know what we most need to use the model I think it's it's kind of creating this feedback loop I say in the sense that you.

Let's say developer and anthropic is like ah it it would be better if it was better at this X thing and then you bake that into the next model that you build that that's that's one version of it but but then there's just the ordinary product iteration of like you know we have a bunch of we have a bunch of coders within anthropic like we.

You know they like use quad code every day and so we get fast feedback that was more important in the early days now of course there are millions of people using it.

And so we get a bunch of external feedback as well but it's you know it's just great to be able to get you know kind of kind of.

Fast fast internal feedback you know I think this is the reason why we launched a coding model and you know didn't launch a pharmaceutical company right it you know you know my backgrounds in.

In my backgrounds in in like biology but like we don't have any of the resources that are needed to launch a pharmaceutical company.

So there's been a ton of hyper on open claw and I wanted to check it out for myself.

I'm going to date coming up this weekend and I don't have anything planned yet so I gave open claw a mercury debit card.

I said a couple hundred dollar limit and I said surprise me.

Okay, so here's the Nakmini it's on and besides having access to my mercury it's totally quarantined.

Now as you felt quite comfortable giving an access to a debit card because mercury makes it super easy to start with guardrails.

I was able to customize permissions, cap the spend and restrict a category of purchases.

I wanted to make sure the debit card worked so I asked open claw to just make a test transaction and decided to do it in a couple bucks to Wikipedia.

Besides that I have no idea what's going to happen.

I will report back on the next episode about how it goes.

In the meantime if you want a personal banking solution that can accommodate all the different ways that people use their money.

Even experimental ones like this one visit mercury.com slash personal mercury is a fintech company not an FDIC insured bank.

Bigging services provided through choice financial group and column NA members FDIC.

You know she thinks we're getting coffee and walking around the neighborhood.

Let me ask you about now making AI go well.

It seems like whatever vision we have about how AI goes well has to be compatible with two things.

One is the ability to build and run AI is diffusing extremely rapidly and two is that the population of AI's the amount we have in their intelligence will also increase very rapidly.

And that means that lots of people will be able to build huge populations of misaligned AI's or AI's which are just like companies which are trying to increase their footprint or have weird psyche is like Sydney Bing but now they're superhuman.

What is a vision for a world in which we have an equilibrium that is compatible with lots of different AI's several which are misaligned running around.

Yeah so I think you know in the adolescence of technology I was kind of you know skeptical of like the balance of power but I think I was particularly skeptical of or The thing I was specifically skeptical of is you have like three or four of these companies like kind of all building models that are kind of dry you know sort of sort of like derived from the like derived from the same thing.

And you know that that these would check each other or even that kind of you know any number of them would would would would would check each other like we might live in a offense dominant world where you know like one person or one AI model is like smart enough to do something that like causes damage for everything else.

I think in the I mean in the short run we have a limited number of players now so we can start by within the limited number of players we.

You know we kind of you know we we need to put in place the you know the safeguards we need to make sure everyone does the right alignment work we need to make sure everyone has bio classifiers like you know those are those are kind of immediate things we need to do.

I agree that you know that that doesn't solve the problem in the long run particularly if the ability of AI models to make other AI models proliferates then you know the whole thing can kind of you know can become.

Harder solve you know I think in the long run we need some architecture of governance right some are some architecture of governance that preserves human freedom but but kind of also allows us to like you know governed the very large number of kind of.

You know a human systems AI systems hybrid hybrid hybrid human human you know hybrid hybrid human AI like you know companies or like or like or like economic unit so you know we're getting to think about like you know how do we how do we protect the world against.

You know bioterrorism how do we protect the world against like you know against like against like mirror life like you know probably we're going to need to you know.

Need some kind of like AI monitoring system that like money you know kind of monitors for for all these things but then we need to build this in a way that like.

You know preserve civil liberties and like our constitutional rights so I think just just as is as is anything else like it's it's like a new security landscape with a new set of.

You know a new set of tools and a new set of vulnerabilities.

And I think my worry is if we had 100 years for this to happen all very slowly we'd get used to it you know like we've gotten used to like you know the presence of you know the presence of explosives in society or like the you know the presence of various.

You know like new weapons or the you know the presence of video cameras we would get used to it over over over 100 and we develop governance mechanisms we'd make our mistakes.

My my worry is just this happening also fast and so I think maybe we need to do our thinking.

Faster about how to make these governance mechanisms work yeah.

It seems like in an offense dominant world.

Over the course of the next century so the idea is making the power set would happen in the next century happen in some period of five to ten years.

But we would still need to see mechanisms or balance of power would be similarly intractable even if humans were the only game in town.

And so I guess we have the advice of AI we.

It fundamentally doesn't seem like a totally different ball game here if Jackson balances were going to work they would work with humans as well if they aren't going to work they wouldn't work with the eyes as well.

And so maybe this is just doom as human checks and balances as well but yeah again I think there's some way to I think there's some way to make this happen like it you know it just it just you know the governments of the world may have to work together to make it happen like.

You know we may have to you may have to talk to AI is about kind of you know building societal structures in such a way that like these these defenses are possible.

I don't know I mean this is so this is you know I don't want to say so far ahead in time but like so far ahead in tech technological ability that may happen over a short period of time.

For us to anticipate an advance speaking of governments getting involved on December 26 the tendency legislature introduced a bill which said quote.

It would be a fence for a person to knowingly train our official intelligence to provide emotional support including through open ended conversations with the user and of course one of the things that.

Claude attempts to do is be a thoughtful thoughtful friend thoughtful knowledgeable friend and in general it seems like we're going to have this patchwork of state laws.

A lot of the benefits that normal people could experience as a result of AI are going to be curtail especially when we get into the kinds of things you discuss in machines of loving grace biological freedom mental health improvements etc etc.

It seems easy to imagine world in which these get whack them all the way by different laws whereas bills like this don't seem to address the actual existential threats that you're concerned about so.

I'm curious about to understand in the context of things like this your anthropic position against the federal moratorium on state a laws yes so I don't know there's there's many different things going on at once right I think I think that that.

I think that particular law is is dumb like you know I think it was it was clearly made by legislators who just probably had little idea what AI models could do would not do they're like hey I model serving as that that just sounds scary like I don't want I don't want that to happen so you know we're we're not we're not in favor of that right but but but that you know that that wasn't the thing that was being voted I think that was being voted on is.

We're going to ban all state regulation of AI for 10 years with no apparent plan to do any federal regulation of AI which would take Congress to pass which is a very high bar so you know the idea that we'd ban states from anything for 10 years.

And people said they had a plan for federal government but you know there was no actual there was no proposal on the table there was no actual attempt.

Given the serious dangers that I lay out in.

Addressence of technology around things like the you know kind of biological weapons and bioterrorism autonomy risk.

And the timelines we've been talking about like 10 years is an eternity like that's that's that's a.

I think that's a crazy thing to do so if if that's the choice if that's what you force us to choose.

Then then we're going to we're going to choose not to have that moratorium and you know that I think that the benefits of that position exceed the costs.

But it's it's not a perfect position if that's the choice now I think the thing that we should do the thing that I would support is the federal government should step in not saying states you can't regulate but here's what we're going to do.

And and states you can't differ from this right like I think preemption is fine in the sense of saying that federal government says here's our standards this applies to everyone states can't do something different that would be something I would support if it will be done in the right way what.

But but this idea of states you can do anything and we're not doing anything either that that struck that struck us as.

Very much not making sense and I think we'll not age well was already starting to not age well with with all the backlash that you see now in terms of in terms of what we would want I mean you know the things we've talked about are starting with transparency standards.

You know in order to monitor some of these autonomy risks and bioterrorism risks as the risk become more serious.

As we as we get more evidence for them then I think we could be more aggressive in some targeted ways and and say hey AI bioterrorism is really a threat.

Let's let's pass a law that kind of forces people to have classifiers and I could even imagine.

It depends depends how serious the threat it ends up being we don't know for sure then we need to pursue this in an intellectually honest way where we say ahead of time.

The risk has not emerged yet but I could certainly imagine with the pace that things are going that you know I could imagine a world where later this year we say hey this this AI bioterrorism stuff is really serious we should do something about it.

We should put it in a federal we should you know put it in a federal standard and if the federal government won't act we should put it in the state state standard I could totally see that.

I'm concerned about a world where.

If you just consider the the pace of progress you're expecting the life cycle of of legislation.

You know that the benefits are as you say because a diffusion lag the benefits are slow enough that I really do think this patchwork of on the current trajectory this patchwork of state laws would prohibit.

I mean having an emotional chat about friend is something that freaks people out then just imagine the kinds of actual benefits from AI we want.

Normal people to be able to experience from improvements in health and health span and improvements in mental health and so forth whereas at the same time.

It seems like you think the dangers are already on the horizon and I just don't see that much.

It seems like would be especially injurious to the benefits of AI as compared to the dangers of AI and so that that's maybe the where the cost benefit makes less sense to me.

So so there's a few things here right I mean people talk about there being thousands of these state laws.

First of all the vast mass majority of them do not pass and you know the you know the world works a certain way in theory but like.

Because the laws have been passed doesn't mean it's really enforced right the people the people you know implementing it may be like oh my god this is stupid it would mean shutting off like you know.

Everything that's ever been built in everything that's ever been built in Tennessee so you know very often laws are interpreted in like.

You know a way that makes them that that that makes them not as dangerous or not as harmful on on the same side of course you have to worry if you're passing the law to stop a bad thing you had this you had this problem as well.

Look my my look I mean my basic view is you know if if if you know we could decide you know what laws were passed and how things were done which you know we're only one small input input into that you know I would.

deregulate a lot of the stuff around the health benefits of AI I think you know I don't worry as much about the like the the kind of chatbot.

Laws I actually worry more about the drug approval process where I think AI models are going to.

Gray league salary the rate at which we discover drugs and just the pipeline will get jammed up like the pipeline will not be prepared to like process all all the stuff that's going through it so.

You know I think I think reform of the regulatory process to buy us more towards we have a lot of things coming where the safety and the efficacy is actually going to be.

Really crisp and clear like I mean a beautiful thing really really really crisp and clear and like really really effective but you know and and maybe we don't need all this all this some.

Like all this superstructure around it that was designed around an era of drugs that barely work and often have serious side effects but at the same time I think we should be ramping up quite significantly the.

You know this this kind of safety and security legislation and you know like I've said you know starting with transparency is is my view of.

Trying not to hamper the industry right trying to find the right balance I'm worried about it some people criticize my essay for saying that's too slow the dangers of AI will come too soon if we do that well basically I kind of think like.

The last six months and maybe the next few months are going to be about transparency and then.

If these if these risks emerge one more certain of them which I think we might be as soon as let as later this year then I think we need to act.

Very fast in the areas that we've actually seen the risk like I think the only way to do this is to be nimble now the legislative processes normally not nimble but we we need to.

Emphasize to everyone involved the urgency of this that's why I'm sending this message of urgency right that's why I wrote.

Adolescence of technology I wanted policy makers to read it I wanted economists to read it I want national security professionals to read it you know I want decision makers to read it so that they have some hope of acting faster than they would have otherwise.

Is there anything you can do or advocate that would.

Make it's more certain that the benefits of AI are.

Our better instantiated where I feel like you have worked with legislators to be like okay we're going to prevent bioterism here away we're going to increase.

Concernancy we're going to increase risk of lower protection and I just think by default the actual bent like the things we're looking forward to here.

It just seems very easy this seem very fragile to different kinds of moral panics or political economy problems.

I don't actually so so I don't actually agree that much in the developed world I feel like.

You know in the developed world like markets function pretty well and when there's when there's like a lot of money to be made on something and it's clearly the best available alternative to actually hard for the regulatory system to stop it.

You know we're seeing that in AI itself right I you know like a thing I've been trying to fight for is export controls on chips to China right and like that's in the national security interests of the US like you know that's like square within the you know the policy beliefs of you know every almost everyone in Congress of both parties but.

And you know I think the cases very clear the counter arguments against it are.

I'll politely call them fishy.

And yet it doesn't happen and we sell the chips because there's there's so much money there's so much money riding on it.

And you know the that money wants to be made and in that case in my opinion that's a bad thing.

But but it also it also applies when it's a good thing and and so I don't think that if we're talking about drugs and benefits of the technology.

I I I I am not as worried about those benefits being hampered in the developed world I am a little worried about them going too slow and I as I said I do think we should work.

To speed the approval process in the FDA I do think we should fight against these chatbot bills that you're describing right described individually I'm against them I think they're stupid.

But I actually think the bigger worry is a developing world.

Where we don't have functioning markets where you know we often can't build on the technology that that we've had I worry more that those folks will get left behind.

And I worry that even if the cures are developed you know maybe there's someone in rural Mississippi who doesn't get it as well right that's a that's a kind of smaller version of the thing the concern we have in the developing world and so the things we've been doing are you know we work with you know we work with you know philanthropists right you know we work with folks.

Who you know who you know deliver you know medicine and health interventions to you know to developing world to sub to here in Africa.

You know India Latin America you know other other developing parts of the world that's the thing I think that won't happen on its own.

You mentioned extra controls yeah why can't the US and China both have a country of geniuses why can't you know why won't it happen or why shouldn't it happen why shouldn't it happen.

You know I think I think if this does happen you know then then we kind of have a.

Well we could have a few situations if we have like an offense dominant situation we could have a situation like nuclear weapons but like more dangerous right where it's like gun.

You know kind of kind of either side could could easily destroy everything.

We could also have a world where it's kind of it's unstable like nuclear equilibrium is stable right because it's you know it's like deterrence.

But let's say there were uncertainty about like if the two AI's fought which AI would win.

That could create instability right you often have conflict when the two sides have a different assessment of their likelihood of winning right if one side is like oh yeah.

There's a 90% chance all win and the other sides like there's a 90% chance all win then then a fight is much more likely.

They can't both be right but they can't both think that.

But this is a completely general argument against the diffusion of AI technology which is the implication of this world.

Let me let me just go because I think we will get diffusion eventually the other concern I have is that people the governments will oppress their own people with AI.

And so you know I'm just I'm worried about some world where you have a country that's already you know kind of you know there's there's a government that kind of already.

You know is is kind of kind of building a you know a tech high tech authoritarian state.

And to be clear about the government there's not about the people like people we need to find a way for people everywhere to benefit.

My worry here is about governments.

So yeah my you know my worry is if the world gets carved up into two pieces one of those two pieces could be authoritarian or totalitarian in the way that's very difficult to displace.

Now will will governments eventually get powerful AI and and you know there's risk of authoritarianism yes will governments eventually get powerfully I and there's risk of.

You know of kind of bad bad bad bad equilibrium.

Yes, I think both things but the initial conditions matter right you know at some point we're need we're going to need to set up the rules of the road.

I'm not saying that one country either the United States or coalition of democracies would be a better set up or requires more international cooperation that we currently seem to want to make.

But you know I don't think a coalition of democracies or certainly one country should just say these are the rules of the road there's going to be some negotiation right the world is going to have to.

And what I would like is that the the you know the democratic nations of the world those with you know who are.

Those governments have represent closer to pro human values are are holding the stronger hand then have have more leverage when the rules of the road are set and and so I'm I'm very concerned about that initial condition.

I am as realistic to interview from three years ago and one of the ways it age poorly is that I kept asking questions assuming there's going to be some key.

Folk or moment two to three years or now when in fact being that far out it just seems like progress continues AI improves AI's more diffuse people use it for more things it seems like you're imagining a world in the future where.

The countries get together and here's the rule of the world and here's the leverage we have here's the leverage you have when it seems like on courage directory everybody will have more AI.

Some of that AI will be used by authoritarian countries some of that within the authoritarian countries will be by private actors versus state actors it's not clear.

Who will benefit more it's always unpredictable to tell an advance you know it seems like the internet privileged authoritarian countries more than you would have expected.

And maybe the AI will be the opposite way around so I want to better understand what you're imagining here yeah yeah so so just to be precise about it.

I think the exponential of the underlying technology will continue as it has before right the models get smarter and smarter even when they get to country of geniuses in the data center you know.

I think you can continue to make the model smarter there's a question of like getting diminishing returns on their value in the world right how much does it matter after you've already solved human biology or you know.

At some point you can do harder math you can do more abstract math problems but nothing after that matters but putting that aside.

I do think the exponential will continue but there will be certain distinguished points on the exponential and companies individuals countries will reach those points at different times.

So you know there's there you know could there be some you know I talk about is a nuclear deterrent still in the lessons of technology is a nuclear deterrent still stable in the world of of a I don't know but that's that's an example of like one thing we've taken for granted that like the technology could reach such a level that it's no longer like you know we can no longer be certain of at least.

You know think of think of others you know there there there there are kind of points where if you if you reach a certain point you may be you have offensive cyber dominance and like every every computer system is transparent you after that.

You must the other side has it has a kind of equivalent defense so I don't know what the critical moment is or if there's a single critical moment but I think there will be either a critical moment a small number of critical moments or some critical window where it's like.

I is AI confer some large advantage from the perspective of national security and one country or coalition has reached it before others that you know that that you know I'm not advocating that they're just like okay we're in charge now that's not that's not how I think about it you know that there's always the other side is catching up there's extreme actions you're not willing to take and and it's not right to take you know to take complete.

To take complete control anyway but but at the point that that happens I think people are going to understand that the world has changed and there there's going to be some.

Negotiation implicit or implicit about what what is the what is the post day I world order look like and and I think my interest is in.

You know making that negotiation be one in which you know classical liberal democracy has you know has a strong hand well I want to show that better means because you say in the essay quote a talk or see is simply not a form of government that people can accept in the post powerfully I age and that sounds that you're saying.

The CCP as an institution cannot exist after we get a GI and that seems like.

A like a very strong demand and it seems to imply a world where the leading lab or the leading country will be able to and by the language should.

Get to determine how the world is governed or what kinds of governments are allowed and not allowed.

So when when I believe that paragraph was I think I said something like you could take it even further and say.

So I wasn't I wasn't necessarily endorsing that that that I wasn't necessarily endorsing that view I you know I was saying like.

Here's if first you know here's a weaker thing that I believe you know I think I said you know we have to worry a lot about authoritarian and you know we should try and.

Kind of kind of check them and limit their power like you could take this kind of further much more interventionist view that says like authoritarian countries with AI are these.

You know these kind of self fulfilling cycles that you can't that are very hard to displace and so you just need to get rid of them from from the beginning that that has exactly all the problems you say which is you know.

If you were to make a commitment to overthrowing every authoritarian country I mean they then they would take a bunch of actions now that like you know that that could could lead to instability so that that.

You know that that that that just that just may not be possible but the point I was making that I do endorse is that it is it is quite possible that you know today.

You know the view or at least my view or the view in most the Western world is is democracy is a better form of government than authoritarianism.

But it's not like if a country's authoritarian we don't react the way we react it if they committed a genocide or something right and and I guess what I'm saying is I'm a little worried that in the age of a GI authoritarianism will have a different meaning it will be a grave or thing.

You have to decide when we are another how how how how how how to deal with that and the interventionist view is one possible view I was exploring such views you know.

You know when you end up being the right view it may end up being too extreme to be the right view but I do have hope and one piece of hope I have is.

There is we have seen that as new technologies are invented.

I mentioned this in adolescence of technology where I said you know like feudalism was basically you know like a form of government right and then when when we invented industrialization.

It's sustainable no longer made sense why is that hope why couldn't that imply that democracy is no longer going to be well competitive systems right it could go it could go either way right but but I actually so I these problems with authoritarianism right that the problems of authoritarianism get deeper.

I just I wonder if that's an indicator of other problems that authoritarianism will have right in other words people become because of authoritarianism becomes worse people are more afraid of authoritarianism they work harder to stop it it's more of a like you have to think in terms of totally equilibrium right.

I just wonder if it will motivate new ways of thinking about with with with the new technology how to preserve and protect freedom and and even more optimistically will it lead to a collective reckoning and you know.

A kind of a more emphatic realization of how important some of the things we take as individual rights are right a more emphatic realization that we just we really can't give these away there's there we've seen there's no other way to live that actually works.

I I I am actually I am actually hopeful that I guess one way to say it it sounds too idealistic but I actually believe it could be the case is that is that dictatorships become morally obsolete they become morally unworkable forms of government.

And that and that and that the the crisis that that creates is is sufficient to force us to find another way.

I think there is genuinely a tough question here which I'm not sure how you resolve and we've had to come out one word another on a through history right so with China and the 70s and 80s have decided even though it's an authoritarian system we will engage with it.

And I think it's right that was the right call because it is state authoritarian system but a billion plus people are much wealthier and better off than they would have otherwise been.

And it's not clear that it would have stopped being an authoritarian country otherwise you can just look at North Korea as an example of that right and I don't know if that to stop that much intelligence to remain in the authoritarian country that.

It continues to co less its own power as you can just imagine a North Korea with any I that's much worse than everybody else is but still enough to keep power and and and then so in general it seems like.

Should we just have this attitude of the benefits of AI will in the form of all these.

Empowerments of humanity and health and so forth will be big and in historically we have decided it's good to spread the benefits of technology widely even with even to people whose governments are authoritarian.

And I think I guess it is a tough question with how to think about it with AI but.

Historically we have said yes this is a positive some world and it's still worth diffusing technology.

Yeah so so there are a number of choices we have I you know I think framing this as.

A kind of government to government decision and you know in national security terms that's like one lens but there are a lot of other lenses like.

I could imagine a world where you know we produce all these curious to diseases and like the you know the the curious to diseases are fine to sell to authoritarian countries the data centers just aren't right the chips and the data centers just aren't.

And and that the AI industry itself you know like like another possibility is and I think folks should think about this like you know could there be.

We can make either that naturally happen as a result of AI or that we could make happen by building technology on AI could we create an equilibrium where where it becomes.

Infeasible for authoritarian countries to deny their people kind of private use of the benefits of the technology.

You know are there are there are there are there are there equilibrium where we can kind of give everyone in authoritarian country their own AI model that kind of.

You know like defend themselves from surveillance and there isn't a way for the authoritarian country to like crack crack down on this while while retaining power.

I don't know that that sounds to me like if that went far enough it would be.

It would be a reason why authoritarian countries would disintegrate from the inside but but maybe there's a middle world where like.

There's an equilibrium where if they want to hold on to power the authoritarian's can't deny kind of individualized access access to the technology.

But I actually do have a hope for the for the for the for the more radical version which is you know is it possible that the technology might.

And here we have properties are that by building on it in certain ways we could create properties that that that that have this kind of dissolving effect on authoritarian structures now.

We we hope originally right we think back to the beginning of the bomb administration we thought originally that that you know social media and the internet would have that property turns out not to but but I don't know what what if we could.

What if we could try again with with the knowledge of how many things could go wrong and that this is a different technology.

I don't know that it would work, but it's worth the try.

Yeah, I think it's a it's very unpredictable like there's first principles reasons why authoritarian is a very unpredictable I don't think I mean we got it we just got it we kind of.

We got to recognize the problem and then we got to come up with 10 things we can try and we got to try those and then assess whether they're working or which ones are working if any and then try new ones if the old ones are.

I guess whether that's how to today is you say we will not sell data centers or sorry chips and then the ability to make chips to China and so in some sense you are denying.

There will be some benefits to that's right the Chinese economy Chinese people etc because we're doing that and then there'll also be benefits to the American economy because.

It's a positive some world we could trade they could have their country data centers doing one thing we could have hours doing another and already we you're saying it's not worth that.

Positive some stipend to empower this country is what what I would say is that you know we are we are about to be in a world where.

Growth and economic value will come very easily if right if we're able to build these powerfully I models growth and economic value will come very easily what will not come easily is distribution of benefits distribution of wealth political freedom.

You know these are the things that are going to be hard to achieve and so when I think about policy I think I think that.

The technology and the market will deliver all the fundamental benefits you know almost almost faster than we can take them.

And and that these questions about about distribution and political freedom and rights are are are the ones that that will actually matter and that policy should focus on.

And it's big and distribution as you're mentioning we have developing countries and.

In many cases catch up growth as we've been weaker than we would have hoped for yes when catch up growth does happen it's fundamentally because.

They have underutilized labor you can bring the capital and know how from developing countries to these countries and then they can grow quite rapidly yes obviously in a world where.

Labor is no longer the constraining factor this mechanism no longer works it's is the hope basically to rely on.

Philanthropy from the people who immediately get wealthy from AI or from the countries that get wealthy.

What is it I mean I mean philanthropy should obviously play some role as it has you know as it has as in the past.

But I think growth is always growth is always better and stronger if we can make it.

So you know what are the relevant industries in like in like in like in like an AI driven world look there's lots of stuff.

You know like there's you know I said I said we shouldn't build data centers in China but there's no reason we shouldn't build data centers in Africa right.

In fact I think it'd be great to build data centers in Africa you know as long as they're not owned by China we should build we should build data centers in Africa I think that's that's that's I think that's a great thing to do.

You know we should also build you know there's no reason we can't build you know a pharmaceutical industry that's like AI driven like you know the the if AI is accelerating accelerating.

Drug discovery then you know there will be a bunch of biotech startups like let's make sure some of those happen in the developing world and certainly during the transition.

I'm moving talk about the point where humans have no role but but humans will have still have some role and starting up these companies and supervising supervising the AI models.

So let's make sure some of those humans are humans in the developing world so that fast growth can happen there as well.

You guys recently announced quad is going to have a constitution that's aligned to set of values and not necessarily just the end user.

And there's a world that can imagine where if it is aligned to the end user it preserves the balance of power we have in the world today because everybody gets to have their own AI that's advocating for them.

And so the ratio of bad actors and good actors stays constant it seems to work out for our world today.

Why is it better not to do that but to have a specific set of values that the AI should carry forward.

Yeah, so I'm not sure I'd quite draw the distinction in that way.

There are maybe two relevant distinctions here which are I think you're talking about a mix of the two like one is should we give them model a set of instructions about do this and versus don't do this.

And the other you know versus should we give the model a set of principles for you know for a kind of how to act.

And and there it's it's you know it's you know it's it's just it's kind of purely a practical and empirical thing that we've observed that by teaching the model principles getting it to learn from principles.

It's behavior is more consistent it's easier to cover edge cases and the model is more likely to do what people want it to do.

In other words if you know if you're like you know don't tell people how to hot wire a car don't speak in Korean don't you know that that you know just you know if you give it a list of rules.

It doesn't really understand the rules and it's kind of hard to generalize from them you know if it's just kind of a like.

You know list of do do's and don'ts words if you give it principles and then you know it has some hard guardrails like don't make biological weapons but overall you're trying to understand.

What it should be aiming to do how it should be aiming to operate so just from a practical perspective that turns out to be just a more effective way to trade in the model.

That's one piece of it so that you know it's the kind of rules versus principles trade off.

Another thing you're talking about which is kind of like the core durability versus like you know I would say kind of intrinsic motivation trade off which is like how much to the model be a kind of.

I don't know like a skin suit or something where you know you know you just kind of you know it just kind of directly follows the instructions that are given to it by whoever is giving it those instructions.

First is how much should the model have an inherent set of values and go off and do things on its own.

And and there I would actually say everything about the model is actually closer to the direction of like you know it should mostly do what people want it should mostly follow these we're not trying to build something that like you know goes off and runs the world on its own.

We're actually pretty far on the coragible side now now what we do say is there are certain things that the model won't do right that it's like you know that that that I think we say it in various ways in the constitution.

That under normal circumstances if someone asks the model to do a task you should do that task that that should be the default.

But if you've asked it to do something dangerous or if you've you know if you've asked it to you know to kind of harm someone else.

Then the model is unwilling to do that so I actually think of it as like a mostly a mostly coragible model that has some limits but those limits are based on principles.

Yeah I mean then the fundamental question is how are those principles determined and this is not a special question for anthropic this would be a question for any property.

But because you have been the ones to actually write down the principles I get to ask you this question.

Normally a constitution is like you write it down and it's set in stone and there's a process of updating it and changing it and so forth.

In this case it seems like a document that people inthropic right that can be changed at any time that guides the behavior of systems are going to be the basis of a lot of economic activity.

What is the how do you think about how how those principles should be set yes.

So I think there's there's two there's maybe three.

Three kind of sizes of loop here right three three ways to iterate one is you can iterate we iterate within thethropic we train the model we're not happy with it and we kind of change the constitution.

And I think that's good to do and you know putting out publicly you know making updates to the constitution everyone's a while saying here is a new constitution right I think that's good to do because people can comment on it.

The second level of loop is different companies will have different constitutions and you know I think it's useful for like anthropic puts out a constitution and you know.

Gemini model puts out a constitution and you know other companies put out a constitution and then they're making kind of look at them compare outside observers can critique and say this this I like this one this thing from this constitution and this thing from that constitution.

And then kind of that that creates some kind of you know soft incentive and feedback for all the companies to like take the best of each elements and improve then I think there's a third loop which is.

You know society beyond the AI companies and beyond just those who kind of you know who who comment on the constitutions without hard power and there you know we've done some experiments like you know a couple years ago we didn't experiment with I think it was called the collective intelligence project to like.

You know to basically pull people and ask them what should be in our AI constitution and and you know.

I think at the time we incorporated some of those changes and so you could imagine with the new approach we've taken to the constitution doing something like that.

It's a little harder because it's like that was actually an easier approach to take when the constitution was like a list of do's and don'ts.

At the level of principle to ask to have a certain amount of coherence.

But you could you could still imagine getting views from a wide variety of people and I think you could also imagine and this is like a crazy idea but hey you know this whole interview is about crazy ideas right so.

You know you could even imagine systems of kind of representative government having having input right like.

You know I wouldn't I wouldn't do this today because a legislative process is so slow like this is exactly why I think we should be careful about the legislative process and AI regulation.

But there's no reason you couldn't in principle say like you know all AI you know all AI models have to have a constitution that starts with like.

These things and then like you can append you can append other things after it but like there has to be this special section that like takes present I wouldn't do that that's too rigid that that sounds.

You know that that that that sounds kind of overly prescriptive in a way that I think overly aggressive legislation is but like that is a thing you could that you know like that is that is a thing you could try to do is there's some.

Much less heavy handed version of that maybe I really like control loop to where obviously this is not how constitutions of actual governments do or short work where there's not this vague sense in which.

The Supreme Court will feel out how people are feeling and where the vibes and then update the of the constitution accordingly.

So there's yeah with actual governments there's a more procedural process exactly but you actually have a vision.

Of competition between constitutions which is actually very reminiscent of how.

Some libertarian charter citizen people used to talk about what an archipelago of different kinds of governments just like and then there would be selection among them of who could operate the most effectively.

Yes, in which place people would be the happiest and in a sense you're actually yeah there's this vision.

I'm kind of recreating that.

Yeah, I think that vision has has you know if things to recommend it and things that things that things that will kind of go wrong with it.

You know I think I think it's a I think it's an interesting in some ways compelling vision but also things will go wrong with it that you hadn't that you hadn't imagined so.

You know I like loop to as well but I feel like the whole thing has got to be some some mix of loops one two and three and it's it's a matter of the proportions right I think that's got to be the answer.

When somebody eventually writes the equivalent of the making of the atomic bomb for this era.

What is the thing that will be hardest to glean for the historical record they're most likely to miss.

I think a few things one is at every moment of this exponential the extent to which the world outside it didn't understand it.

This is this is a bias that's often present in history where anything that actually happened looks inevitable in retrospect and and so you know I think when people.

When people look back it will be hard for them to put themselves in the place of.

People who are actually making a bet on this thing to happen that.

Wasn't inevitable that we had these arguments like the arguments that I make for scaling or that continual learning will be solved.

You know that that.

Some of us internally in our heads put a high probability on this happening but but it's like there's there's a world outside us that's not that's not acting on it's not kind of not acting on that at all.

And and I think I think the the weirdness of it.

I think unfortunately like the insularity of it like you know.

If we're one year or two years away from it happening like the average person on the street has no idea and that's one of the things I'm trying to change like with the memos with talking to policymakers.

But like I don't know I think I think that's just a that's just like a crazy that's just like a crazy thing.

Finally I would say and and this probably applies to almost all historical moments of crisis.

How absolutely fast it was happening how everything was happening all at once and so decisions that you might think.

You know we're kind of carefully calculated.

Well actually you have to make that decision and then you have to make 30 other decisions on the on the same day because it's all happening so fast and and you don't even know which decisions are going to turn out to be consequential so.

One of my one of my I guess worries although it's also an insight into into you know into kind of what's happening is that you know some very critical decision will be will be some decision that you know someone just comes into my office and is like.

You have two minutes like you know should we should we do you know should we do thing thing A or thing B on this like you know someone gives me this random you know half page half page memo and it's like should we should we do A or B and I'm like I don't know I have to launch let's do B and and you know that ends up being the most consequential thing effort.

So final question.

It seems like you have.

There's not tech CEOs who are usually writing 50 page memos every few months and it seems like you have managed to build a rule for yourself and accompanying around you which is compatible with this more.

Intellectual type roll SEO and I want to understand how you construct that and how like how does that work to be.

You just go away for a couple of weeks and then you tell your company this is the memo like here's what we're doing.

It's also reported you read a bunch of these internally.

Yeah so I mean for this particular one you know I wrote it over winter break.

So that was the type you know and I was having a hard time finding the time to actually find it to actually write it.

But I actually think about this in a broader way.

I actually think it relates to the culture of the company.

So I probably spend a third maybe 40% of my time making sure the culture of anthropic is good.

As anthropic has gotten larger it's gotten harder to just you know get involved in like you know directly involved in like the train of the models, the launch of the models.

The building of the products like it's 2500 people it's like you know there's just you know I have certain instincts but like there's only you know.

It's very difficult to get to get involved in every single detail you know I like I try as much as possible.

But one thing that's very leverage is making sure anthropic is a good place to work people like working there.

Everyone thinks that themselves as team members have one works together instead of against each other.

And you know we've seen as some of the other AI companies have grown without naming any names.

You know we're starting to see decoherence and people fighting each other and you know I would argue there was even a lot of that from the beginning but but you know that it's it's gotten worse.

But I think we've done an extraordinarily good job even if not perfect of holding the company together.

Making everyone feel the mission that we're sincere about the mission and that you know everyone has faith that everyone else there is working for the right reason that we're a team that people aren't trying to get ahead of each other's expense or backstab each other which again.

And it happens a lot at some of the other places.

And and how do you make that the case I mean it's a lot of things you know it's me it's it's it's Danielle who you know runs the company day to day.

It's the co-founders it's the other people we hire it's the environment try to create but I think an important thing in the culture is.

I some and just you know so the you know the other leaders as well but especially me half to articulate what the company is about.

Why it's doing what it's doing what it's strategy is what its values are what its mission is and what it stands for and you know when you get to 2500 people you can't do that person by person you have to write or you have to speak to the whole company.

This is why I get up in front of the whole company every two weeks and speak for an hour.

It's actually I mean I wouldn't say I write essays internally I do two things one I write this thing called a dvq.

Dario vision quest.

I wasn't the one who named it that that's the name it it received and it's one of these names that I kind of I tried to fight it because it made it sound like I was like going off and smoking.

And smoking pale years.

But but the name just stuck so I get up in front of the company every two weeks I have like a three or four page document and I just kind of talk through like three or four different topics about what's going on internally.

The you know the models were producing the products the outside industry the world as a whole as it relates to AI and geopolitically in general you know just some mix of that.

And I just go through very very honestly I just go through when I just I just say you know this is this is what I'm thinking this is what anthropically leadership is thinking and then I answer questions and and that direct connection.

I think has a lot of value that is hard to achieve when you're passing things down the chain you know six six levels deep.

And you know a large fraction of the company comes comes to attend either either in person or either in person or virtually and it you know it really means that you can communicate a lot.

And then the other thing I do is I just you know I have a channel and slack where I just write a bunch of things and comment a lot.

And often that's in response to you know just things I'm seeing at the company or questions people ask or like.

You know we do internal surveys and there are things people are concerned about and so I'll write them up and I'm like I'm you know I'm I'm just I'm very honest about these things you know I just I just say them very directly.

And the point is to get a reputation of telling the company the truth about what's happening to call things what they are to acknowledge problems to avoid the sort of corpo speak.

The kind of defensive communication that often is necessary in public because you know the world is very large and full of people who are you know.

But you know if you have a company of people who you trust and we try to hire people that we trust then then you know you can you can you know you can you can really just be entirely unfiltered.

And you know I think I think that's an enormous strength of the company makes it a better place to work and makes people more you know more of the sum of their parts and increases like we accomplish the mission because everyone is on the same page about the mission.

The beginning discussing how best to accomplish the mission.

Well, in lieu of an external Dario vision quest, we have this interview.

This interview is a little like that.

This is in front of you.

Thanks for doing it.

Yeah, thank you Dr. Cash.

Hey everybody, I hope you enjoyed that episode.

If you did the most helpful thing you can do is just share it with other people who you think might enjoy.

It's also helpful if you leave a rating or a comment on whatever platform you're listening on.

If you're interested in sponsoring the podcast, you can reach out at barcash.com slash advertise.

Otherwise, I'll see you at the next one.