AI Radar

Noteworthy AI conversations and writing, summarized from publisher notes.

Noam Brown on Why Traditional Benchmarks Fail Modern AI Models

· No Priors

OpenAI research scientist Noam Brown argues that standard benchmark grids are misleading because they fail to control for test-time compute. He illustrates this with the GPT-5.5 release, where benchmark scores showed only modest improvements over 5.4, yet users found 5.5 substantially better because it was far more efficient with its thinking time. He proposes that evaluations should either fix a token/cost/time budget or plot performance as a function of test-time compute, rather than reporting single numbers. A central concern is that responsible scaling policies and preparedness frameworks were designed before test-time compute became important, so they don't account for the fact that a model's capability is now a function of the budget spent on inference. A model that looks benign at $10 of compute could be far more capable at $10,000 or $10 million. Combined with model release cycles of 2-3 months, Brown argues labs cannot fully characterize their models' ceilings before the next one ships, and external users are systematically under-exploring what's already released (e.g., 5.5 could likely have disproven the Erdős unit distance conjecture with $100K of compute). On the trajectory question, Brown pushes back on the idea of an overnight intelligence explosion, arguing that heavy reliance on test-time compute makes wall-clock time a fundamental bottleneck. He sees AI research taste as the current limiting factor and views multi-agent coordination as a still-underdeveloped frontier. On competition between labs, he describes an intensely competitive but broadly aligned research community. He closes by encouraging the industry to break out of the "bad equilibrium" of publishing uninformative benchmark grids and to embrace budget-controlled evaluations, including for routing-based systems.

AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.