AI Solves Open Problems in Theoretical Physics — Alex Lupsasca, OpenAI

Source material

Transcript

Okay, so I think we're at the special time now where at least it's some directions AI has become superhuman at least on certain tasks and that's what led to these recent papers that resolve a problem that was puzzling physicists experts in the field for over a year and they were unable to resolve it and AI was able to do it very quickly.

So I think that's a certain milestone that we've passed by the you guys are bringing attention to this because I think maybe for the average person on the street who doesn't care about theoretical physics this is not very noticeable but I think it's a very profound change and we've really passed some kind of a threshold.

Welcome to the A for Science podcast a part of Lean Space Network.

I'm Brandon, I develop our nature of ptx using AI at the topic AI.

I'm joined by my co-host, RJHonicy, CTO and founder of Neuroomics.

Yeah, it's a pleasure to introduce Alex Luchoska, professor at Vanderbilt University and fellow at Open AI.

He has it for a young researcher and he has quite a story background amongst other things.

He's the winner of the 2021 New Horizons Breakthrough Prize.

It's the quality of the Oscars for Science.

I asked chat to UT as the most prestigious award someone of his career could win and it recommended a second one called the IUPAP Award which turns out to you also won.

Anyway right now he's having fun at Open AI doing some really cool research of pushing the foundation of theoretical physics using GPT models.

A pleasure to be here.

The one message I wanted to convey is that I think we're on this trajectory which I personally find very surprising and yeah, honey surreal but also amazing where I would say a little over a year ago AI was very useful for email but not the kind of work that I do that I consider important theoretical physics calculations.

I thought that's special much harder than email and AI is not going to be able to do that.

Then there were a series of developments that came in rapid succession that completely changed my mind and I can walk you through some of these examples specifically in particular Chagipiti O3 was the first really strong reasoning model that could do actual math that was useful for my research and could save me a lot of time.

That's when I started to be a bit of tension and use it a lot more and I thought, wow this is a great tool.

I get ahead of this and learn how to integrate into my workflow then when GPT 5 came out it was able to reproduce one of my best papers that took me a very long time to come up with in like 30 minutes and that's when I really became AI pill.

I thought, oh my god this changes everything it's the most important discovery in my lifetime it's going to affect everything about how we do research and frankly a lot of my colleagues I would go around telling them this currently I think I have to be tensioned and yeah I was getting lots of different reactions but I think people weren't quite getting it but I talked to open AI they were also really excited and I thought I don't know that much about AI but I have to get in on this and to understand that this is happening and not be a part of it is a huge mistake so I have to go to open AI so I was on sabbatical it was very easy to come here and join the company and then it just kept ramping up even beyond that and you know to the point where now I think most of my senior colleagues in physics are aware of where things are ahead of and are all getting on board so yeah I think that's awesome sorry sorry I was just like I find it really funny that you that story because it almost reminds me of a lot of different people who had the same realization with codex starting sometime last fall especially it just took off and a bunch of people are like even like Andre Carfathi wouldn't from oh man this is you know 20% of my work it's kind of a nice you know assistant to oh crap what just happened well yeah in August actually I remember when GPT-5 came out that point I was really following AI pretty closely and I think on Twitter the reception was lukewarm a lot of people like well we expected a lot more and it's not better writing email and I remember thinking well okay GPT-3 could write email how much better can it get writing email that's not the point but the science frontier the capabilities were really taking off yeah there was a lot of attention I think paid even to oh three but yeah but then Virginia before five was yeah a huge jump and I think five point four is also a huge jump I don't know how noticeable it is on the outside or that I did hear some I saw some chatter on online people are running these independent benchmarks which do show this so I think people are realizing and also anyway in practice researchers are now all over AI using it and yeah I'm getting in balance all the time because I'm the resident scientist doing physics at OpenAI and so everybody is sending the papers chats oh my god this happened I got one just this week somebody said Codex just wrote up a simulation of the S.Y.K.

Mall this is like a very technical thing in quantum mechanics and gravity and like yeah a lot of research groups have been trying to run this simulation and it couldn't do it and Codex did it in 10 minutes just because setting it up was so hard well I think but partly it's because of the Van Dyagram where you look at the people who have the physics knowledge and the people who have the top coding skills and maybe the overlap it's not that that large although I think it's it's been growing but I think in this example there are a lot of really good people in physics with coding skills would be trying to simulate these things so I think Codex is just really good now okay yeah um nice okay so I think we're at this special time now where at least in some directions AI has become superhuman at least on certain tasks and that's what led to these recent papers which maybe we should talk about that resolve a problem that was puzzling physicists for experts in the field for over a year and they were able to resolve it and AI was able to do it very quickly so I think that's a certain milestone that we've passed and I'm glad that you guys are bringing attention to this because I think maybe for the average person on the street who doesn't care about theoretical physics this is not very noticeable but I think it's a very profound change and we've really passed some kind of a threshold specifically focus on the glue on paper in the physics part and we can get to the AI part later okay so in physics there are two basic principles of nature that we think every law should respect or every theory should respect on the one hand there's the principle of relativity which at some very high level declares there's an absolute law that cannot be broken which is that you cannot transmit information faster than the speed of light but then there's another principle which is the uncertainty principle that underlies quantum canics which says that everything's a little fuzzy yeah or a position velocity there's a little fuzziness to that and so you can see immediately at this level of description already there's attention between these two principles because one is an absolute law declaring you cannot go fastness be light and the other one is saying it's a little bit fuzzy and this is just to give a sense of how when you try to write down these principles in mathematics the equations don't really play nicely with each other and so it's been a real struggle to come up with physical theories that can reconcile simultaneously both principles to describe the physical world around this and I would say that the great achievement of 20th century physics which is really one of the greatest triumphs in in human thought as far as I'm concerned is the elaboration of this framework called quantum field theory which is a general framework that can describe the physical forces of nature in a way that accommodates both of these principles and in quantum field theory which is our best theory today I would say gets a little bit technical but again try to keep it pretty high level what you're trying to compute or describe are the probabilities for certain events to occur because you're in this quantum mechanical setting you can't say with certainty what's going to happen when you have a certain experiment but you want to predict probability distributions and in quantum mechanics probably distributions are obtained by squaring some complex quantities and by complex that I don't mean complicated I mean they're not real numbers they're they're in that real plus imaginary compass which we call quantum amplitudes so the goal of theory is to predict quantum amplitudes which are these objects complex objects that square the quantum probabilities and that's the most you can say about the outcome of an experiment and these quantum amplitudes in particular there's a variety of them called scattering amplitudes which describe the following scenario suppose you have a bunch of particles that you throw at one another this is what happens in particle colliders like the LHC of certain in Geneva so you take a bunch of particles you smash them together stuff happens they interact via the physical laws of nature various processes occur and then other particles come out as a result at the end of the interaction and so scattering amplitude is the object that describes the the probability for a particular type of attraction we have some particles coming in with some energies and momentum and some other particles coming out with other energies and momentum and so the scattering amplitudes they're functions of all the data describing the particles coming in and the particles coming out so in general you can have arbitrarily many particles involved in interaction and this is one of the hallmarks of quantum field theory that particles can be destroyed so you don't have the same number of particles at the end necessarily as you have in the beginning particles can be created lots of things can happen and in general you want to describe all the possibilities and so you want to have an amplitude for an arbitrary number n of particles so that's called an n point amplitude because there's n particles coming in out and it turns out in quantum field theory that if you have a particular force and you're able to compute the n point amplitudes these functions of the n parameters of the functions that squirt the probabilities then you know everything about the theory more or less there's always a master's but it's basically the entire content of the theory so yeah if you have a theory that tells you any number of particles come in and go out then I can say I can declare anything about that system exactly then you know everything and importantly these amplitudes they're not just numbers they're functions because the probabilities that they compute depend on how much energy do the particles have what are their momentum and also a particle has something called a lot of particles like the photon which is the particle of light has a polarization so when you look at the surface of a lake and you have polarized sunglasses and you turn your head you can see more or less sunlight reflect it off of the lake and that's because a photon which you can think of as a little particle of light as it propagates it carries a little arrow perpendicular to the direction of propagation which is called the polarization and this polarization has a direction and sunglasses can selectively let in light with one polarization and not the other and this polarization actually as light travels it can rotate it can wind it can do it's own thing and in general if it winds in the right-handed way so as the particle travels if the polarization winds to the right we call that a positive velocity or a right-handed polarization and if it winds in the other direction we call that a left-handed velocity or or negative velocity so in general these amplitudes which are the fundamental object in quantum field theory that we want to contain all the information there is to know about physical forces these amplitudes depend on not just the energies and momentum but also the polarizations now I've told you about how there's two basic principles of nature relativity in quantum mechanics they come together in this framework quantum field theory and I keep talking about forces so there's four fundamental forces of nature there's electromagnetism which is responsible for basically the properties of atomic elements and the periodic table and therefore chemistry and biology and and everything you see touch feel pretty much as call due to electromagnetism textures colors and this force is mediated by the photon which is the particle of light that one is the most familiar to us then there's gravity which is another force that we feel very much because it keeps us to the ground and then there's two nuclear forces the week and the strong nuclear force which we don't really notice directly in our daily lives but the week nuclear force is responsible for radioactive decay and other such processes and the strong force which is the strongest of them all is what binds the new clearest together so you're learning high school that like charges repel but if so then why do protons stick together inside the nucleus of the atom they should repeal one another and indeed that's the case but if you bring in really close then the strong force kicks in and overwhelms the relatively weaker electromagnetic force so the as strong force is mediated by the exchange of the particles of the strong force charcoal gluons because they're what glues together the new place of the atom so gluons are the particles of the strong force and gravity is mediated by gravitons I think like the gluon paper I think was sort of the maybe the starting point for this maybe not but like the gluon paper had like a really like specific result right like yeah absolutely so maybe let me just flash the the paper itself so we put this on the archive a little over a month ago now and here's the here's the paper let me explain in a few sentences now that I've given a lot of background what the title means yeah so the title says single minus gluontree amplitudes are non zero this might self forbid egg but I think we can attack this for the audience so gluons are the particles that carry the strong force and gluon amplitudes are functions that describe the quantum probabilities for gluons to interact via the strong force now the word tree here is a little bit of a technicality it means we're only considering processes where no gluons are created or destroyed if gluons are created or destroyed then you get loops which we can explain later but this is just a technicality so we're considering special interactions where the same gluons that come in also come out so for anyone who's ever fit a polynomial you can think of trees being like a linear term and then loops can be high order characteristics exactly a way more complicated in that but conceptually it's like kind of the lowest order in a series and so single minus now I have to explain that so remember I told you earlier how particle set polarizations so when you try to study gluon amplitudes this is like a whole industry of physics you know it's a very complicated field people have ridden thousands of papers over the decades so you always want to try to understand the simplest examples first that's what you start with the tree amplitudes where the the leading effects and then you foray about the loop corrections so you might think that the simplest example to start with is one in which all the particles have the same velocity so say they're all right handed or that is to say they're all plus velocity particles it's been known for a long time that actually in that case the amplitude is just zero which means the interaction is forbidden and cannot happen that's one way time it's just a cemetery just explicitly for bits of this it's you don't even have to calculate anything you just know yeah just the mental analysis yes a very general arguments yeah you don't need to do very much work and so yeah it's true that it's the simplest example but it's so simple that nothing happens yeah so okay the it's just pretty old you might ask what about the next level up what about it oh no I want to understand you have like a bunch of gluons they're coming into an interaction yeah they're all in the same velocity yeah and then you're just saying that just can't happen yeah okay like because like I can I take my gluongun and shoot and he takes his gluongun and they go there and then that just can't but it's going right through each other oh so they just won't interact they won't interact oh yeah yeah yeah yeah yeah that's a good clarification yeah and now you might ask what if one of them has the opposite velocity but all the others have plus velocity but one of them it has a minus velocity so that's what we would call a single minus amplitude and if you look at the lecture notes and textbooks that have been written on this the same argument that rules out the all plus amplitudes also appears to rule out the single minus amplitudes they're too simple they can't really interact nothing to see here move on so then you might ask okay well what about the next thing where there's two particles that are minus velocity and all the others if so if there's any of them there's n minus two others that have positive velocity all right so these would be double minus amplitudes and people in the 80s studied I've computed these amplitudes they're not zero and in particular there were two physicists Park and Taylor who found this beautiful result they did a lot of really hard work and completely these amplitudes very technical difficult calculation but at the end you get all these terms and you have to sum them all up and almost all of them cancel and at the end you you're left with this very simple formula that fits in half a line which is now known as the Park Taylor formula for these amplitudes and these amplitudes are now called MHV amplitudes which stands for maximally velocity violating because they have the largest or so we thought possibly symmetry between the plus and the minus velocity particle that the most asymmetry now let's get to this paper which came out last month so this is a paper written with Alfredo Gavara who's a postdoc at the Institute for Advanced Study David Skinner, a professor at Cambridge University Andrew Stromminger, a professor at Harvard used to be my advisor and also Kevin Wheel who studied as a particle physicist in a previous life so how did this happen well maybe we'll get into how I ended up in opening I a little bit later but I ended up in opening I started to improve the models abilities to do physics the models got really really good at physics and I thought okay it's so good now we should try to solve some actual research problems at the frontier and I called up Andy who used to be my advisor I said hey Andy do you want to come here to the SF visit opening I and we can try to solve one of your problems in physics and I thought you know it's probably not going to work but if it doesn't work at least we'll figure out why it doesn't work and you know I can do this with a different physicist every month and eventually something will in the meantime will learn how to improve the models so it's all fun and useful and so Andy was the first one that I invited to do this and he said well I have this perfect problem that I've been thinking about with Alfredo and David for the past year I'll explain now the problem but the amazing thing is that we decided to start working on it using AI a little bit before Andy was scheduled to come like the week before and in fact using Chad GBT we solved the problem before he even got off the plane which was a huge surprise to him and to me to be honest I had not expected that and it's really cool story so Andy David and Alfredo understood a year ago that this statement that the single minus amplitudes the statement that there is zero is not exactly correct because the usual argument in the lecture notes and textbooks has a loophole and the loophole is that it assumes that the particles are coming from generic directions but in a certain regime where the particles are exactly aligned with one another we say they're collinear then the usual argument has a loophole and it's possible for the amplitudes to not be zero but then if they're not zero what are they so that suddenly these really simple amplitudes previously thought to be zero if they're not zero we should compute them and they should be something really nice and simple and special now I'm bearing a lot of I'm sweeping a lot of details on the drug here but this has to work in some different signature space time you know it connects lots of other things they've been working about we're not going to worry about this I mean I was actually hoping at the end maybe we could talk about what it means to be two dimensions in space and two dimensions in time but yeah I mean I think like part of this is doable the loophole is one about you know the alignment of the particles but it's also a loophole about the space time of physics that universe we're living in and this is not the really violent stuff so um the understood that they're not zero and they started to compute them and Alfredo is really I think the unsigned hero the story because he did a lot of really hard work to confuse these things by hand and I'll just show you an example so we in the paper there's a lot of formalism so here is the beginning of the definition of the general answer one yes it's very hard to unpack but it starts here then you have to define these vertices objects V and they're complicated involve sign and theta functions of spinners and then you have this recursive formula by okay it's a whole mess and concretely if you try to unpack this definition remember these amplitudes are a function of the number of particles involved so there's a three-point amplitude there's only three blue ones in the interaction and you know the answer is pretty simple this is some function that we've defined here not that complicated then this is the four-point amplitude when now there's four particles and you can see that we go from one term to a sum of two terms here but then once you get to five particles you get a you start to get a lot more terms there's eight of them being summed here and by the time you get to six terms for those people not watching this on YouTube and listening this equation takes up a quarter of the page is 32 terms each of which is a product of four terms each of which is itself encapsulating a rather complicated formula yeah so this is super nasty and that's as far as Alfredo got or anyone so it's so I don't know is this just an expansion of some sort of how hard is it to do this expansion very hard okay yeah and there's a nice graphical way to understand this in terms of Feynman diagrams I hadn't planned to explain this but there's a visual this is kind of a visual subject so the math is very complicated and already back in the 40s Richard Feynman who's one of the pioneers of quantum field theory came up with this very visual way to organize our understanding of the subjects you can do to all these little cartoons that represent possible interactions and the rules of quantum mechanics actually say that in these amplitudes where you scatter a bunch of particles you get to fix what comes in and what comes out because that that's the question you're asking what's the probability for a certain traction but then everything that happens in between you don't get to choose that because the physical laws determine what happens and actually in quantum mechanics you're supposed to consider all the possibilities all the ways in which the incoming particles can interact and transform into the outgoing particles and you're supposed to average or sum over all the possibilities to get the final amplitude for the process as a sum over the amplitudes for each individual possibility for how do you could get there so just to be clear there's incoming particles they interact and then there's all these different they each have their own amplitudes and then it's sort of like I select for this one one possibility and this one one possibility and then I get like one possible interaction and then there's an infinite number of those for each and then I sum those infinite last both so and I get that out yeah so in principle there are infinitely many pictures to sum over but that's why we organize them by how complex they are and it turns out that every time you get an interaction every time there's a that the vertex where it lines meet that point interaction comes with a power of the coupling constant which controls the strength of the interaction and it turns out that every additional interaction makes the amplitude more suppressed so it it's contributes less to the final answer and so you want to first consider the diagrams with a fewest possible number of attractions because they will give you most of the total final amplitude and then if you're trying to get a more and more refined answer you then consider the more and more complicated cartoons with more and more interactions and in fact this is one of the ways in which the diagrams can it complicate is that they can have loops so for instance here you have a particle that decays into two particles creating the slope because then they meet it up again and disappear so in this interaction you have intermediate particles being created and destroyed but whenever that happens you get two extra vertices in your graph so these diagrams are suppressed because it's less likely to happen that you get these extras solicitors interactions and so you have to you don't need to worry about this as much it's like a small in correction and of course in principle you can keep going but you're never done it's as in very special circumstances.

I heard her powers in a polynomial or something right or a Taylor series and so to go back to the story back in 80s with an image of the amplitudes which I think now is a bit of a misnomer I would call them double-minus amplitudes because that's where we're going to get to in a second right there was this heroic calculation where a lot of Feynman diagrams were summed and they were considering more and more interactions with more and more particles and every time there were more and more terms but they all canceled that the end always give a simple answer and in fact that's what this PT term PT stands for Park Taylor these formulas are you know they fit in the line so it's not that complicated but it's very surprising that such a messy calculation at the end will clean up into such a simple result and so what Alfredo and the end of it did was to understand that these single-minus amplitudes in the special case where some of the particles are aligned they don't have to be zero and then you can do this very complicated Feynman diagram expansion to get the answer which is not zero but the problem is if you do it this way while you can represent the answer in some horrendous horrendously messy complicated way but if you unpack it it's extremely complicated it's complex in the following sense when you consider the endpoint amplitude so the probability of end particles interacting the number of terms in your answer which correspond to the number of diagrams roughly that you have to add up it grows effectively in end the number of particles and factorial growth is really bad it's super exponential it grows faster than an exponential so it blows up in your phase this is what you're seeing here and that's because roughly you have to draw all the possible cartoons and the possible combinations is a combinatorial problem and that's where the factorial behavior comes from but we know from the 80s that in the actually more complicated double minus case park and Taylor found this miraculous simplification and so Andy Alfredo and David spent the last year chasing the analog of the park Taylor formula the very simple answer that was obtained in the 80s for the double-minus amplitudes but now for these single-minus amplitudes which are not zero but then what what are they and they were getting this really complicated answer and okay you never know in physics ahead of time if something will simplify you have to believe in it to find this simplification but because the double-minus wants to simplify it felt like these should simplify too and we think they're reporting for lots of things and these are somehow really important objects that are very fundamental so they should have a nice description and so they spent a year looking for that there's a funny the next line if you scroll down it's the thing like we need a simpler formula right so whatever we need a mark is easy yet more precise formula is needed and this is where AI comes in because when I asked Andy hey do you have a problem in your pocket that we should use AI to target he said well I have just the perfect thing for you we've been positive about this it's really important it's really interesting it connects to all these things and we don't know the answer I mean like when I was a grad student if I had a approach something like this I would have probably played it into a computer algebra system chucked along tried a few limit cases see if there's any who will need like magical you know simplifications which happen um this type of thing is you know something that oftentimes you see this you're like we need a different approach exactly how then um before I'd even got here we started to play with Chad Gpt and Alfredo Andy and I were trying different things lots of different chats happening go back and forth they did as well and the first thing that happened is that we fed the five point amplitude into Chad Gpt and we're like can you simplify this and it's like you know there's a special region so there's an extra assumption that you can make in which this answer simplifies to this one so this assumption is equivalent to the you have one particle coming in and it decays into in minus one that's one way to think about it rough okay yeah but we're in two time dimension so yeah it's complicated but basically you can look at what we call face base it's the entire space of possibilities for all the energies of incoming particles and the momentum and there's a special region in that face base where one particle has one different sign of it's frequency compared to the other and in that region there's a big simplification that happens the Chad Gpt found and I should say it's this was the public model but the pro version of that things really hard so it was at a unknown fact that it just was able to relate to the problem or was at something that it put together with as far as I know it looked that together okay it said you know this five point function which is a sum of eight terms each one of which is a product of three terms they're all pretty complicated it said hey like actually this simplifies to this product of only three terms and we stare at this thing thought wow that's really nice we didn't know this it's actually in hindsight once you know you can redrive this but it takes a while to understand where this comes from so I think that was a leap of inside the DEI had and I think what it did I mean at some point said I wrote a Python code and I ran through all 5,000 possibilities and I okay I did you stand in this so it's the equivalent of running his computer algebra system but it just decided to do it on its own and came up with a huge simplification so great yes this was after making the assumption this is after the the K one particle the K assumption yes so it made it figured out there was a lot of exchange this is a very experimental we're talking about a lot of figure out there's some four region in which things simplified and that region it said okay this thing simplifies yeah or CBT came up with that simplification as well yeah yeah and then we're like okay well let's give it the six point function which I'll write it perfectly computed and by we didn't have the set of point function we I don't think anybody can use the identity to expand it it would be disgusting and then Chad GPD doesn't slow thing and then it's like oh yep simplifies to this and we thought whoa okay yeah that is really nice because now instead of 32 terms it reduces to just four terms and it's not as sum of 32 terms it's a product of only four terms and then we asked Chad GPD okay well can you guess the general formula for all n and that's step by the way I mean you could imagine using some programming language or symbolic manipulation software to do these reductions in certain examples but to tackle the general case I don't know how to use computer to do that but Chad GPD said yeah this is the answer in the general case boom I'll fix that take you know it's like using pro with things for 20 minutes at a time you go back it wasn't like six days or something no no it's just like over the course of of several interactions and the amazing thing is that the formula that it proposed instead of having this factorial growth which is super exponential where the number of terms as you consider n the number n of an increasing number n of particles the number of terms blows up here it's actually linear so if you double the number of particles you only double the number of terms it's the nicest possible behavior could imagine this is the equivalent I think of the part Taylor formula for the double minus amplitudes that was known back in the 80s but now for the single minus amplitudes and this was guessed by GPD I think was 5.2 at the time GPD 5.2 Pro but it couldn't quite derive it so I said it looks like this but I don't know how to prove that yeah I think the model was not quite strong it off okay prove it but part of my work it open and I has been to develop stronger physics capabilities in the models and a lot of people have been adding lots of you know it's not just my single contribution there's a lot of great research happening and all comes together you know it takes a village but we had this internal model that could think for a very long time and was x was strong at physics so we gave it the whole problem from scratch without actually giving it this we just formulated the problem in a very sharp way and asked the model to solve to find the answer for the amplitude in the general case in this region because now we've identified that this was the special place to look and it took 12 hours which is a long time but it came back with the same formula which we had not given it so it rediscovered the correct formula but this time it also found the proof that the formula is correct the drive that and in fact the remainder of the paper after we state the equation is devoted to the proof that is basically what came out of the AI so we say the rest of this work is devoted to proving that the conjecture is correct there's three steps first you show this second you fill blind and third you show blind and then this is basically what the AI came up with so now I can finally summarize the paper the title is single minus gluontri amplitudes are non-zero so these special interactions between gluons were only one of them has a different velocity from the others which were previously thought to never occur actually these interactions can happen so the amplitudes are non-zero that's the main claim of the paper I think it's quite surprising I think it's the it could really nice paper and the final result I guess there's two results one is understanding that it's not zero that came from the humans like a year ago but they were trying really really hard to find this simple answer for what the amplitude is and they were kind of stumped for a year they were able to get this indirect representation that's extremely complicated in terms of Feynman diagrams but they were looking for this simple formula that is the analog of this part the lower work from the 80s for the more complicated amplitudes and that was done with the AI and so I I think that's a really interesting result yes amazing changes the way you should think about where we are at physics and how AI is going to change that you know it's not just hype I mean this is like a real thing they happen it's a result that top researchers in this field were thinking about for a year and then the AI solved it so I think that's interesting there's several things about the story which I think people didn't understand on Twitter if maybe it's called down to like equation 38 to what's it 35 to the 38 yeah like so I would say most even intro grad students would look at 35 to 38 and say 39 is actually very natural extension of this like yeah that is I don't think you know that surprising I think it's interesting I didn't know until just now that you can that when you prove 39 that was a fresh session that was without the lending cases you started from scratch yes yeah did you do it that way because I guess it's an extra way to be confident in the answer if the if a different model and it then only comes up with it from scratch then you're not just feeding spoon feeding the answer that you think is correct that's the next or confirmation but yeah I think we thought a lot about how to put this out into the world and there's no perfect way to do this I you know clearly we could have done a better job of communicating it one thing that was important to us is to not make this paper about AI because I think this is a really interesting physics result you know people will keep reading this paper I hope for a lot of time we didn't put AI in the abstract because this is a physics result that stands on its own there's one paragraph really about AI where we just say the the final formula was first contracture to be a GPT 5.2 pro and then proved by the total pinnacle because you know that's what happened it's true but we didn't really want to get into it because I think that's not the point of the paper I mean it's really interesting how it happened but the result stands on its own and I think if you read a paper today there was written 20 years ago that used the computer to do some critical step in the argument and it had this whole discussion of how well I loaded MSDOS 3.1 and it had 5 floppy disks and I had to swap my floppy disks you wouldn't care you don't know why you're reading the physics paper today so we didn't really want to go into that in the paper and we talked a little bit about it in the the blog post that we released with the open AI which is this one and then I guess on Twitter there were a lot of questions and I wrote some tweets that I think clarified it and there was somebody who is a physicist who wrote a great blog post like actually understanding the story and the economist also put out a great article about it which they really understood what happened and I thought it was a great great coverage science magazine also read about it harvon the institute for a man study put out press releases so I think it got a lot of attention but it's kind of a subtle thing to explain it took us an hour to go through what happened and what was done so you know it's hard to explain I think it would have been kind of a distraction from the physics point of the paper to go into that okay let's talk about the physics and give us a sense because you know my theoretical physics on the frontier it comes from pbs based time right like I'm you know it's a great channel yeah and gives you a great high level picture but hard to know how this sits in the pantheon of papers that represent the cutting edge of theoretical for you're asking me how good is the paper not exactly I want to just understand like it seems like you're comparing it to this previous result that is pretty significant and like you know like highly cited and very important how does this compare to that okay you're putting me in a bit of a tough spot I will say I think the result is surprising that's why the title is what it is you know single minus simple here there are not zero and if you're if you're somebody who works in this field that's your catch retention ultimately it's very hard to know in science when you release something into the world how it's going to be received and how impactfully it will be I think the true value of a paper can only be assessed test of theories into the future based on how much future work it leads to and what developments it opens up so maybe a better way of asking is so my understanding is that previous paper kind of opened up a whole line of thinking about yeah I think this is a great segue too the second paper that came out just three breaks later in fact then let's talk so it got its own blog post this is March 4th so I guess two weeks ago now so we're talking earlier about how there's four forces strong force mediated by google ons and then gravity that's mediated by gravitons except google ons we can produce a DLC we can measure their effects fairly directly gravitons we think are also around those big produced all the time we even as I move my hands but we've never done an experiment that directly measures gravitons but they're supposed to be the quantum of gravity so they're really interested from a theoretical standpoint and so we're going back to RJ's question earlier what is a graviton these different answers we could give ultimately the correct answer depends on what the theory of quantum gravity which we don't know yeah if we just naively try to take all of your tricks from field theory that we know from the standard model apply to gravity things just break down the theory is not self-consistence there is some definition of various problems yeah just like if you took in this room there's light flowing around there's some indivisible bit of light that you at some point can break up into smaller bits that's the quantum of light we call that the photon and the gravitational force is mediated by the exchange of gravitational force or gravitational waves if you try to take a gravitational wave and break it up into small and swallow pieces at some point you get a quantum that you can't break up anymore and that would be the graviton that's how we understand them so there's the idea being like you can't you get to a certain point and you can't have less gravity than that you either have some or not right that's one way to think yeah and so we wrote this paper which is called single minus graviton tri-epletus or non-zero so it's the same title almost except with graviton instead of a glue on and that's all purpose because we we wanted to extend the result and it's the same story in the sense that it would start at all single minus amplitudes or zero but actually it's not true and also for for gravity but gravity is a lot more complicated so now if you want to compute the graviton amplitudes it's potentially a lot harder do gravitons have face the same way that the glons do so they actually have spin two rather than spin one is getting it to the amount of the numbers you have to use to describe them are a little bit different they're doubled in some sense okay so their polarization is more complicated I see but this is really getting it to the wings but the special region in which the final answer simplifies has two labels because it's a spin two particle whereas in the glue on case there was only one label because it was a spin one particle so this is like a Greek so it's not the same math glue on the graviton do have some spiritual similarities compared to other types of particles play yes in the sense of their particles of force yeah yeah but they're like sort of doubled yeah this sort of doubled down yeah I'd be okay this is I guess the the people watching this podcast probably like to geek out of this so the modern definition of a particle in quantum field theory which is our best verified framework for nature is that particles are irreducible representations of the pucorate group we just lost 90% of our own yeah okay well maybe we cut this so there's mathematical representations and they've all been classified all the possibilities are known by vignor actually a bring physicist and it turns out that the representations are possible particles are completely labeled by the mass and the spin and the charge of the particles so these are the three quantum numbers and particles of long-range forces like gravity and electromagnetism have zero mass they have to have integer spin and spin one is three of the four forces and spin two is gravity and then that's it but let's set that aside the really cool thing about this paper is that what first of all it came out three weeks after the first one which is really fast and I think this is a great example of AI accelerating science and in fact we could have put this paper out three days after the first one because that's that's how fast we got the answer out of Chagipiti but it took us three weeks because we wanted to check very carefully that it's correct but most of the time was spent verifying the answer not dreading which is insane actually if you take a step back it shoots if you told me a year ago yeah like you're going to have this AI that just those really hard calculations for you and then most of the human effort goes to verifying the answer I thought that you know you crazy so this is very surreal and then we also had to write it up as a nice paper which you know put in the citations and references that they take some time and also had a baby in the meantime so it's a lost time there but we do this really fast so I think it's an example of excited science another really cool thing is that for this paper we didn't have to use an internal open AI model that had to think for hours this was all done using the publicly available GPT Pro in fact we shared one of the main prompts that we used it's if you go to the blockpost extending single minus amplitudes to gravitons and you scroll down to the text there's a link to one of the chats that we used so you can see we used Chagipiti 5.2 Pro and the amazing thing about this is that we gave it the glue on paper as a seed and we said we can understand the paper make sure you understand the manipulations in the appendices because that's where most of the hardware goes and but it comes back and it says yep um I have to the paper let me focus on the appendices here's what happened and basically the punchline is that GPT Pro with the glue on paper as an anchor was able to do the graviton calculation which is really different mathematically completely out of its own from well I guess not from scratch from the glue on paper but it's it's just a different thing and it was strong enough to do it completely so you're the papers and I took the conceptual leap from the paper the previous paper and and just said okay what math do I need to make that same concept and it's different math that's an important thing to emphasize so in particular there's a crucial application of something called the directed matrix tree theorem and Alfredo and David we've been thinking about these for a very long time we're like oh that's really cool that's surprising we hadn't thought of that or you've seen that before that was like no math but it applied was because maybe it has such broad understanding of math and physics that it's able to say oh this is what this is a good thing to apply in this case yeah exactly and so here it understood the paper the glue on one and then we said okay well the task is to generalize this paper to the gravity case here's some here are two key changes but otherwise manipulation should be similar so we tweak some things at the get-go and then we said good luck you're a brilliant theorist so it's like a you know we give it to paragraph so we give it to google on paper a couple paragraphs it's a good luck thought for 20 minutes and boom it starts the thing so it starts at the beginning it works through the implications oh like really interesting stuff and then it says here's what I would do next to turn this into the gravity paper if you want I can do block and so we said yeah go ahead so another thought for 31 minutes yeah so this this this exchange is 110 pages but I think it's hilarious I would describe this as vibe physics because you can see a scenario goes away there's a lot of hard work goes lots of equations it's starting to do the focus and I have to use this different math you have to use these tree calculations LSC reduction formulas okay there's a lot of stuff happening subwoofer trees concrete checks it's starting to yeah well this is one of the things I love is that it's able to do the same things that a human would do which is check some basic cases a study check and to get intuition and so it comes back every three minutes so well here's what remains to to finish the full gravity paper and there's a list if you want I can write the gravity yes do that this is the first that okay it goes back thanks for 34 minutes half collineer support let's start the best that okay this these formulas actually made in the paper in some form this is all correct there's a bunch of stuff at the end it says if you want the most the next most useful thing I could do is do this and we're like yeah I have verified this by performing the explicit check and it goes on it just to cut to the end finally just we say okay right up the paper and you can see the paper that it writes out and it's very close to the final thing that we actually put on the archive so did it make suggestions that were not what you would have suggested as the next steps it's very smart it knows kind of where to go it's useful to steer it if you compared what it came up with with the actual paper that we put in the intro the abstract and introduction were written by Andy who's an amazing writer and I think he gave this wider perspective on the problem and how it fits into physics and how it connects to other things that you know the idea didn't do it just the intro it wrote was more generic but okay I could write really well we didn't really try to make it yeah and the other thing is we added the section this section two which was not part of that initial exchange is about how these graviton amplitudes transform under certain symmetries of physics and that's something that we're really really interested in because we eventually want to understand quantum gravity as I mentioned earlier and typically the first step to uncovering a new theory is to understand what are its symmetries that's something that gives you some kind of ground to stand on and the in particular has been pushing this program of celestial holography which is like a whole thing we could get into but it's an exploration of the symmetries of quantum gravity and he really wanted to understand this and there's a separate chat we didn't share that one or where we let the AI to explain all these answers fit into the symmetries that we know the theory should have and that's something they went in there but actually I think from section three onwards it's pretty much very close to what the AI wrote so I would say this is really remarkable it's a real solid result in quantum gravity that was done pretty much completely by the AI with human steering it and asking kind of the right questions but all the math was derived by by Chagipiti Pro the public model you can access and most of the time spent was by by as the humans was like checking everything and writing it up and that's really wild I mean we're really so I mean that you've as a physicist you find yourself where a lot of coders have found themselves where there's a kind of a fundamental maybe a epistemological question here that if now as a physicist like I could have done that maybe maybe like I needed a little more background but like a lot of it was like yeah go ahead like take this paper give it some prompt you guys obviously prompted very well but there wasn't like maybe a undergraduate in physics could have come up with a lot of that and so the question is how does the undergraduate in physics now learn when they don't have to do the hard calculation similar to how does they'll undergraduate coder actually you're you're opening up many different strands of conversation shows super interesting so let's try to unpack that a little bit so the most direct thing you asked is how does the next generation learn yeah that's a really good question I think about this a lot and now that a lot of senior physicists in the field are coming to grips with these new capabilities one of the questions that comes up very quickly is how do we train the next generation because the way we were trained is by going through these you know there's these difficult rights of passage where you have to do these really arduous calculations and this is how you build confidence in your own abilities and check test your knowledge and it's not just about what you're capable of doing it's about knowing that you're capable of doing it and proving it to yourself and building that self confidence it is important and we don't have a good answer this is something that academia is going to have to grapple with so one thing that is especially difficult is that as a professor I have graduate students and the the gap between where classes take you even graduate courses they only go so far they go very far but only so far and the gap between where that ends and research begins is actually huge and it's growing wider usually is a professor what you do is when you take on your students you keep in your pocket a few easy problems in this sense that you know they're going to work some questions that you you know in principle you could work out not that difficult but you give them to a students so that they go through the exercise of learning everything around the question and develop in the technology and then you know enough about the problem that you're sure there's an answer that you can get there and you can advise the student in the process of discovering it and I think the issue is that many such problems now I would say these models can probably cross yeah these are probably we usually take again you know time scale for theoretical physics papers six months to year that's pretty typical so if you tell a student go and think for six months about this one question and you have to work really hard to learn a lot of stuff around it and do lots of calculations even the most determined students would they not within the course of six months ever ask that's a little bit weird now it's also an opportunity because I remember that time in my graduate school career in my second year of grad school I took all my graduate courses my first year and then my second year was my first project and it was actually the hardest time for me in graduate school to traverse the desert for more classes to take you to the research frontier it's very hard and this is a lot of time spent banging your head against the wall like all the time your confused you don't understand things just because you need to absorb so much knowledge and AI can totally help you with that yeah it's the best teacher it knows everything it can unpack any complicated fact to any desired level of detail actually my experience is a trained professional physicist working on my own research using GPT now is that I would say there's two key ways in which my research is completely changed one is that I spend much less time being confused so I'll do a calculation get an answer and I think how does this fit in with this other fact that I know like how do I reconcile these things in my mind I'm confused how I do that all that time yeah in research usually you you you take a step then you're like you had a roadblock and obstacle you're confused and you have to think for a few days maybe you go for a walk or you work on another project come back get a new idea but you spend a lot of time confused that's day to research with GPT I'm like hey I just did this I found this how does this match for this other thing and then it's like oh well you forgot this thing or oh you didn't quite think about it correctly or does the standard fact you know it's so the amount of time you spend confused just dramatically shrinks and you move so much faster that's one of the accelerating effects the other accelerating effect is that you know I only have so much free time and energy especially when you become a professor you have to teach you have students you have grants to the mystery there's a lot of things you have to do so you're free time to think about research without distractions shrinks and also you know you only have so much energy to do hard calculations and so what you would usually do is if you have a problem you know you're at point A and you want to get to point C you think about the route oh I have to go through point B first and actually maybe the multiple points and you try to plot in your mind the course that you're going to take before you actually go start to the hard work you try to dig really hard about where you're going into the charter course with AI actually you can launch ten instances of chat and have each one try a different route and send it as a scout that moves very fast into the unknown pushing out words and you can just very quickly get some feedback to see oh okay these approaches are not promising these are are much more promising and then if you follow them there's a huge difference between being the first to push into the unknown versus following someone ahead of you and even if chat GPT doesn't always get everything right just kind of having a scout that signposts some key steps along the way that you can use to anchor your own movement is extremely helpful so this is like two concrete ways that AI has changed the way I work and I think if you're entering research having an assistant that can help you find your way to where you're trying to go can be very good so I think it's it's inevitably going to change how we work how we operate and how we train students and you know part of what's exciting about my job is trying to figure how it works but it's not just a job for open eye it's actually a job for every research or a professor more generally to think about this I think the future is very bright because we have some challenges to overcome but on balance this is such an amazing tool I think it's going to give human physicists AI superpowers because of what I just described you can do so much more and I think actually the kind of skill that is really useful to get great results out of AI is very similar to the kind of skill that you develop as an academic collaborating with other humans this is like a collaborator and if you're a professor it's been advising students and postdocs you know a lot of what being a professor involves is knowing for each student postdoc that you're working with exactly what question to give them so matching the problem to the person and knowing how to give them the question in a way with how much what level of detail not too much not too little and that's actually kind of what you have to think about when you interact with chat GPT so I think it's a transferable skill and at people who are good at this are about to get AI superpowers what you just described there reminds me of several conversations we've had on the podcast thus far which key coming up to this concept of taste one of the things that especially you say theoretical physics how did you physics has maybe had a problem with I'm not sure if you want to try it that way but it is you can be very trending that like there's certain things which we come in fashion because maybe right now we're a world where we don't have the data to define a new directions to really guide or constrain where we're going I'm curious like how does essentially something which is superhuman and that it has basically all known physics and is interact with a field where at its core what oftentimes can be popular people start working on is more based upon general aesthetics or what you know the community collectively thinks is cool at the time because I can imagine Iku could vibe so many different worlds you know like for example just using client space using this sort of two time two spatial dimensions for this was an already sort of an assumption that I think is actually kind of important in some ways and does provide feedback to our world but in the concept of you know you could have asked chatgbt to solve this problem in all sorts of number of ways and you know maybe it could come up with all sorts of things which don't really align with maybe the useful taste as a community how do you actually deal with that like a proliferation of really interesting results but it's actually not clear what is where the field should go.

You're getting at the heart of what does it mean to do progress and be a theoretical physics research.

We're doing this a hard question and there is a simple answer if there were it would be research.

Let me say a couple of things.

The first one is when you go to graduate school in physics it's usually because you're really interested in the big questions why are there three dimensions of space what happened at the big band what's inside of black hole.

The things the other things that you know I was thinking about because of sci-fi movies and books that are called you know what you realize is that actually these questions even though they're really cool and exciting they're not really the most fruitful scientific questions because at any given time there's an edge of knowledge and the role of scientists is to extend the edge of knowledge is to push into the unknown and to do that you want to find the questions that are right at the edge or just beyond the edge but not so far that you can't grapple with them.

So the question of why there are three dimensions of space that's a really cool question but I don't know if anyone who said anything really compelling about that so it's just a question that's beyond the edge so it's not as a professional physicist I don't spend my time thinking about this because I just don't know any pathway to solving this question it's not useful to think about so really the process of training as a physicist involves coming to grips with what the edge of knowledge is because that's where the interesting fruitful questions to make progress on is the scientists that's where they live.

Oftentimes when you go through graduate studies you worry oh my god I have to learn about Feynman diagrams and all this math and all these calculational methods and it's true that's a really hard thing to learn.

It takes a lot of work but in some sense you know what's it become a professional physicist you should feel like you can learn any tool you can pick up any tool that is needed for the task at happy and you should develop that confidence and that's what makes a competent physicist.

The competence and physicist is one that can alert any new mathematical tool or you know a piece of code or whatever that is needed to solve the problem that happened and that makes you a good physicist to a competent one if you pick up the skip and in graduate school you know it's daunting you have to learn a lot but by the end you should have a lot of skills in the toolkit and the confidence to pick up any new one is needed.

The difference between a good physicist and a great physicist is knowing what is the right question to ask.

That is actually the hardest part of being a scientist that's knowing what is the next fruitful question to tackle and I think AI right now is a very good physicist.

In fact maybe superhuman would it comes to certain computations but it's like this extremely technically skilled graduate student that you can give a sharp while post question to it will do incredibly hard calculations correctly now and come back to you with the answer.

So it's super competent but one of the things that it doesn't quite have yet is knowing what is the right question to ask and I think just like with humans that is actually the hardest skill to pick up that's the one that comes last.

Yeah I know you're not working directly on the AI so much.

I mean I don't know I exactly how much you but do you get a sense for you know you can imagine a future where you just do better reinforcement learning maybe you change the architecture of the model completely so that it's some other like whatever not transformer and the trajectory just keeps going like this because it's been very very rapid increase since 01 of the capabilities or you get a sense that we're getting near the edge of the frontier of knowledge now so that the sort of the the ability of the model to recombine knowledge in somewhat novel ways is you know like that's kind of in the it seems like not not to disc or like not to play down any of these results but that it seems like there was a lot of what it did and maybe there's some not like this but a lot of what it did was recombination of known facts.

Okay yeah but do you get a sense that that's given you reasonably that will continue or if we're gonna just like sort of okay we know how to recombine stuff really well and we can't push beyond that.

Without getting too full it's hard for me yeah I'm not sure that any of us are anything more than recombination that's right machine.

Working with GPD Pro on this problem the me feels like working with a creative collaborator.

It did stuff that I didn't know that I found surprising and so I think I'm not sure there's a qualitative difference I think it's just a matter of the degree yeah okay that as we continue scaling the capabilities which is certainly happening I don't see why it's gonna stop like we definitely have a bunch of things in the pipeline that are gonna keep coming this year and you know my horizon for seeing it to the future is like not that good but beyond the year but like definitely we're gonna keep scaling up this year and I don't say any reason why it's gonna stop and I think that's gonna make these models display feeds of insight that looked to us like real creativity I would say this already happened in this project at least you know what is creative insight is the end of the eye of the beholder right how we out for go right yeah I want to come up with moves it were very I talked to Terry Tau a couple of weeks ago at UCLA we had an opening eye events with iPam which is this Institute of mathematics there and I talked to Terry Tau and he said that in his view all of the proofs that he's seen AI come up with and math even the ones that are first seen creative and surprising later on were tracked down and found to have really pull facts out of some obscure reference so I don't want to put words in his mouth with my understanding was that you know Terry Tau was not yet been impressed by a creative move in math but you know Terry Tau is a unique individual I've been impressed I consider myself you know my bar is lower and I think as we keep scaling this up I can't go into the details but there's a lot of effort at opening eye is a lot of really smart hard working people that are pushing very hard to to take the next step and I think it's going to come eventually I mean just look at the trajectory that we're on so a year ago I was black hole physicists in academia not really paying too much attention to AI I thought AI's cool for emails but it's not going to do what I do which is special 03 which was the really first really strong reason a model came out and was able to do a calculation for me that would have taken me days and did it 11 minutes and I thought wow that was shocking to me and I could not we could go into the details if we have time I could show you the example because I yeah saved and it was really surprising to me and then I thought okay I got to really start using this tool there's no other software that could do this kind of calculation as far as I know was really surprising and really cool and then GPT 5 came out six months later and that was able to reproduce what of my hardest calculations I think the number of people in the world that could do that you could come on your hands so when you see reproducing this has been published or not published it was a success or internal was so last summer in June I put out this paper which I really like in which I found it's called Why Is There No Love in Black Hole yeah and love is actually a technical term it refers to Augustus love a British mathematicians who study the tides so when you have an object like the moon going around the earth it exerts title forces on the oceans and so you can measure the title response say of the earth and its oceans to the moon why are some coefficients that encode the strength of the title response and these are called love numbers in reference to Augustus love but famously black holes do not experience tides so they have no love and there's been a resurgence of interest in this fact in the last five years because people understood that this can be connected to symmetry principle so in physics whenever something is zero like why should black holes never experience tides that's surprising well oftentimes the answer is because there's a symmetry principle at work that forbids the existence of tides that protects the structure of the black hole and so I found these new symmetries so these are differential operators that act on solutions to this equation that describe perturbations of a black hole and these generators are symmetries because if you act on the solution to this equation you get a new solution you know I thought this was very beautiful I liked it very much and it came out in June on the archive and in August GPT five came out and the cutoff date for its training set proceeds the release of this paper so GPT did not see this paper in training and when it came out I was like okay I'm gonna got to meet Marc Chan who's chief research officer at open AI and he said give GPT pro a really hard problem let's see how good it is and I'm like you want a part problem okay I'm just so this problem and I wrote a paper I was very excited about it I thought this is really deep and cool and I gave GPT the equation here and I said what are the symmetries I didn't tell it to their symmetries because by the folding goofy I should be there aren't any and thought for five minutes and said yeah there are no symmetries which is what usually happens and that was wrong at Marc Chan was visibly stressful it's like oh well okay why don't you give it an easy rare question and so then I gave it the same question but not for a black hole spacetime but for an empty flat spacetime which is a simpler problem but that's actually how I approach this problem myself you know you warm up on the easier question first so I give it the flat space question which is in this paper also so it's it's this equation which looks much simpler then this also has three symmetry generators which are shown here this is not you it's these equations have been studied for 200 years everything in flat space has been known forever and GPT five pro thought for it was like nine minutes and it came up with the answer very beautiful answer perfectly structured perfectly correct actually at the time I I also tried the other models from our competitors and none of them could could get this at the time so GPT pros really ahead and I think it continues to be the best model for this kind of mathematical and physics work and Marc Chan was like okay well this is great but now that it's done the warmer problem in the same instance of chat tried the full problem again now that it's been prime I thought okay why not and so I gave it the same question as before what are the symmetries of this equation now the full black hole problem and this time I thought for 18 minutes which I'd never seen before and it came up with the answer so basically in under 30 minutes with one hand which is the obvious warmer problem to prime the model on first it completely solved this problem which you know is one of the nicest calculations that I've ever done and that really blew my mind that was my move 37 to 7 months yeah so that's how we call it in the air world and once I saw that I thought okay we're on this crazy trajectory where you know 18 months ago was not useful a year ago it could do really hard calculations that would take me days eight months ago it could reproduce some of my best work in like under 30 minutes and then now last in the last month it solved these questions that we've discussed at length which the world experts had spent a year thinking about without being able to get to the answer so you know I think it's just going to keep getting better where we're going to be in six months or a year I don't see any reason why it would stop and I think we're going to be having a very exciting year okay going back to this late these thoughts about scientific discovery and what can these models do versus just being very good at superhuman at solving physics the people keep asking this question hypothetically could we train a version of chatGVT where it's never seen anything post you know 1904 and could it rediscover relativity I think that there's a very analogous question we could ask right here which is new conceptual result about a single minus glue on amplitudes which was sparked by human insight and you know there were some certain very specific assumptions which went in here like understanding working in curr space time is something that people thinking about and has some useful like transferable insight um people have been thinking about you know maximally velocity violating amplitudes for quite some time have you ever tried kind of using a model right before the cut-off date of this and asked given a curr metric is there anything interesting with regards to kylicity violation or maybe turning around saying you know it's long been thought every it's long been known that with the exception of some sort of measure zero due to what there's no you know single minus non zero amplitudes have you tried either of these directions and asked it to like discover a new insight push the boundary as you were just talking about and make a leap in addition to not just solve a problem in human give it but actually could you get the intuition yes yeah you will try this not exactly the counterfactual version that you describing I first haven't done that but pushing the models at the frontier to try to make this type of leap is something that we're very focused on yeah and I think um well okay I don't want to talk about the trail research we're doing but I can say something publicly I think which is you can take this page of this paper and you can feed it to charge your pt pro say I like the best model we had out right now and you can ask it what should I do next give me the top three follow-up questions to ask based on this paper I've done this experiment and the top three questions it comes up with are like my top three questions for what I should do next so I think the models are smart enough now and have enough background knowledge that you know for this paper at sage pt's about as good as me at finding the next date to ask and so that's really interesting and it opens up a lot of so you just you know what is the what is the name of the loop that the agent loop there one is talking about where you just say like okay what's the next question go ahead and solve that what's the next question go ahead so like I guess in this goes back to the question I was asking before which is if you do that and probably has been you've tried it or someone it opening it has tried that like I presume you get to some plateau right where you like you're not pushing the boundary of knowledge anymore or is it just like the plateau is money and if you had more money you could go further yeah just to be very explicit because I haven't said this quite out loud I think we now have models that can really turn out papers that are as good as human written papers in fact this is a bit of a problem because when a professional physicist uses this tool and they steer the model and they check the answer they can get amazing results but there are also people that feed it kind of wrong questions that go off the depend and they submit that to archive and this is a problem that the academic community is dealing is it's try come to grips with now which is this problem AI slot for science this is something we have to figure out but you know I would say that with proper steering you could probably turn out a paper a day now you know just I don't know it's like give the question to Chadji BT it'll solve it if it's not that hard of a question or it's a similar calculation to stuff that's already been done it can totally do it in 30 minutes and then you could say write it up as a paper and you can send it to archive okay so I think that we're already in this moment we've we've passed that threshold this is the new reality and more and more people are catching out at this all the time and so some of them are doing this and this is why the archive is now inundated with with submissions so what's the correct response to this I think we put out these two papers in very fast succession we could spend the rest of the year writing 30 more papers like this I don't think that's what we should be doing instead I think now that we have this new tool they give us a superpowers I think we should just raise the bar for what it means to write a good paper like we should aim higher basically one thing that I'm excited about is that I think these single minus amplitudes papers they open the way now to hold the direction of research which I think is a line of attack on really interesting questions in quantum gravity to go back to the start of the the the sessions is the missing piece of the puzzle fundamental theoretical physics and I think we have a pretty clear line of attack through a series of questions all of which I think will be amenable to solution with AI and so I think you know I'm excited to spend a good part of of this year trying to follow this path but really solve harder and harder problems and you know this is this is a this paper gave an answer to a question that had stomps and the Alfredo and David who are experts in this for a year but we haven't seen an AI yet solve a question that stomps an entire community of physicists for decades that hasn't happened yet but I think given the trajectory that we're on at some point hopefully not too far the future we should see that and so I think that's the exciting thing to try to go towards which is pushing the envelope of our can be done we want to start asking all yes question which is if you could remove one bottleneck for your domain so in this case maybe it's AI for physics or maybe it's physics or maybe it's mostly yeah but if you could remove one bottleneck for four year domain what would that be in why well after top of my head you know I spent so much of my time writing papers and the way I think now is so far from papers it just feels like not the right way somehow to store and communicate knowledge I think an extreme version of this which makes the problem more apparent is math especially certain parts of math where papers are very tourists and they take four pages I had this experience when I was learning algebraic geometry and graduate school going to a mathematician and say what's going on in this like four-page paper it's just like very tourist notation and he said oh forget what's in the paper and he took me to the black morning strategy draw pictures you know it's like this is how you should think about it and I was like oh wow this is amazing but like one of that is in the paper and you know mathematicians I think have this cultural norm that they kind of hide the messy work and they were right these beautiful short pristine papers it depends on the subfield but oftentimes that's the case and that the way they actually think about the subject is a living breathing and today is very different from the way which is recorded in papers some of that is also true for physics you know I love doing calculations coming up with questions finding the answer and then I would say the huge bottle neck is writing it up so somehow it feels like papers are not quite the way of the future or at least the way that we currently operate with I write it up it's sent it to a journal at the six months I don't know it's just like why are we doing all of this feels like maybe there should be something better I mean you could you know if you want to understand this paper one thing you can do is upload it into chat GPT and ask it to explain it to you and you can keep unfolding the complexity into more and more detailed explanations and so if we move into a world where we use AI to do the calculation get the result then we have the step can that's again to a paper and then you know I send the paper to Brandon and he puts it back into an AI that we'll ever get to do it like why are we doing this yes right that's a little bit funny set I feel like you know if you ask me would I be confident that in 20 years we'll have these sort of like static documents in which we publish our results as papers I would think not like that doesn't seem like the best thing we could be doing maybe some kind of interactive paper which lives in some LLM maybe your whole paper is some chat GPT page and you know you there's a chat but attached to the paper and you can say explain the big picture and like zoom into this fact I think we're gonna head in that direction that would be a cool thing to see writing a paper though is a useful exercise because it forces you to condense your thoughts and make them really clear so I'm not saying it's a bad thing to do in general but just the way we do it is is very slow but that's that's the first thing they came to my maybe an another answer is in this project the graviton paper we got to a draft of a paper extremely fast and then we spent most of our time checking the answer so I think that will effectively be big like maybe the next big bottleneck and that is one of the things that the models I would say if you ask me what are what is missing in the models like what can we really improve for scientific research I think we've kind of touched on the two big things already but just to spell them out one is creativity and the spark of invention and really taking the next step I think that will come as we scale at the intelligence well we'll see but I don't know that there's something missing inherently I think it's just it's starting to to make these leaps for me but maybe we should encourage the models to to try to make bigger leaps because largely with models after all they're trained to give you the middle of the road answer like if you ask any I like said Gpt write me an email about blood you wanted to give you kind of the expected answer not to sample from the tales like you know you kind of wanted to give you a reasonable thing so for most tasks you want that but for science research sometimes you want the idea that comes out of left field the thinking outside the box or really sampling far out of the distribution and that's something we could do in principle but that's not how the models are you know we're not really favoring that so we might have to do tweaks of this kind to make the models be able to take bigger leaps and then the second thing is verification because we're now in this new regime where the models are so capable that for very hard computations of the frontier of knowledge they can just do the whole thing but you know is it correct in this case it was correct you know sometimes I get emails from people saying oh it did this really long calculation but those mistakes somewhere disappointed okay I mean the calculators are getting more and more complicated and longer but yeah sometimes they mess up and so I think improving verification or even just having the model indicate more directly how confident it is in its answer because I think they're smart enough to know what they're very confident in the answer versus when they're just kind of guessing in some step and getting the I to be more explicit about that is I think a way to improve them for research and that verification step I think it was going to become maybe a bigger bottleneck this year yeah creating a hum from axiom would agree with you in fact she formal verification is there thing right yeah I think you know it's interesting a year ago I would have said it's super important to have formal verification then the models got so smart that I thought well you know if Brendan I talk about a mathematical proof and we go over it we're not going to formalize it in Sethioretic notation or you know we don't think the way lean which is this language for formal verification reasons we reason through the proof and natural language we use words and so if a model is really smart enough then it should be able to do the same thing and that's we've been saying this huge increase in capability for mathematical reasoning and and developers for using natural language so then maybe for while it's looked like that that wasn't the thing to really focus on but now that we're this regime where you can just get Chagipiti to tackle thousands of questions at the same time and it will return proofs versus the significant fraction of them now actually the onus is back on the humans to verify all the outputs and so yeah is that becomes a bottleneck I think formalizing math and automating verification will become more valuable it looks like to me and that's something we're thinking a lot about as well thanks uh what do you want the audience to take away from today because they're one message that you want them to leave with yeah I think it's important to get the word out um that the model that we're developing at OpenAI or becoming really capable of scientific research I myself was a bit of an AI skeptic a year plus ago because I thought the models are very good at writing tasks but not mathematical tasks that change with O3 the the first strong reasoning models and then GPT5 was able to do some of the hardest calculations that I can do and reproduce them correctly and now recently in the past month we've seen models solve open questions and theoretical physics and now they're solving problems in quantum gravity and quantum field theory so if you just extrapolate that into the future imagine where we're going to be in six months or a year I think it's kind of surreal to live through this time but it's really happening it's really basic and I think we're going to see a lot of big changes happening in research so that's yep it's actually through the space that's they too it's awesome so thank you so much for taking a time this is like I learned a lot from our discussion and and I'm going to definitely keep up with what you're going to thank you it's been great to be here thank you