Inside the Race to Measure Frontier Intelligence
This discussion features Vals founder and CEO Rayan Krishnan, who addresses the increasing difficulty of measuring AI model improvement. He argues for independent, continuously evolving evaluations as public benchmarks saturate and models optimize for tests. The conversation covers why self-reported model scores can be misleading and how Vals evaluates models pre-release. It also explores the importance of evaluations for enterprises to understand AI's return on investment, the implications of token spend rivaling employee salaries, and how evaluations could provide a shared language for model routing, recursive self-improvement, AI policy, and international coordination. The discussion also touches on the geopolitics of evaluations and whose values become embedded.
Original sources
YouTube a16z
AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.