OpenAI on the End of SWE-Bench Verified and Future of Coding Benchmarks
In this episode of Latent Space, Mia Glaese and Olivia Watkins from OpenAI's Frontier Evals and Human Data teams discuss the retirement of the SWE-Bench Verified coding benchmark. They explain that SWE-Bench Verified, once a key benchmark for measuring coding progress, has become saturated and contaminated, limiting its usefulness for tracking improvements in AI coding capabilities. OpenAI is advocating for the community to move towards more challenging and realistic benchmarks like SWE-Bench Pro, which feature longer, more complex tasks and reduced contamination. The guests highlight the extensive human effort involved in curating SWE-Bench Verified, including multiple expert reviews to ensure task fairness. They also discuss the challenges of contamination from training data leakage and overly narrow test criteria that do not fairly assess model coding ability. Looking forward, OpenAI emphasizes the need for benchmarks that measure open-ended design decisions, maintainability, and real-world coding workflows, potentially incorporating human-in-the-loop evaluation and proxy models for grading. The conversation also touches on OpenAI's preparedness framework, which tracks frontier risks including research automation and model autonomy, with coding benchmarks playing a key role. The guests call for more community collaboration in creating and sharing diverse, high-quality evaluations to better measure AI progress and real-world impact, including metrics on AI usage and job augmentation.
Original sources
Podcast Latent Space
Episode:⚡️The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data ↗
YouTube Latent Space
Video:The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals ↗
AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.