AI Radar

Noteworthy AI conversations and writing, summarized from publisher notes.

How Claude 3.7 Plays Pokémon

· Latent Space

David Hershey from Anthropic joins Latent Space to discuss "Claude Plays Pokémon," an agent harness he built to test Claude's long-horizon task abilities by having it play Pokémon Red. He started the project in June 2024 with Claude 3.5 Sonnet and has iterated through every subsequent model release, using Pokémon as both a nostalgic obsession and a pragmatic benchmark for agent capabilities. The architecture is intentionally simple: a tool-using agent with three tools (button presses, a knowledge base for long-term memory, and a Navigator tool that patches Claude's spatial navigation weaknesses), rolling out a conversation of ~30 messages before summarization, peaking around 100K tokens per turn. A core insight Hershey shares is that as models improve, he's been *deleting* prompt instructions rather than adding them—the best gains come from letting the model use its own intuitions rather than imposing the builder's. Claude 3.7 Sonnet showed clear improvements (it finally beat Brock, caught Pokémon, and navigated out of the lab), but vision and spatial reasoning remain glaring weaknesses—it sometimes walks into walls for 12 hours straight. Hershey sees Pokémon as a useful, if expensive (thousands of dollars in tokens), qualitative proxy for agent progress, and is excited to see how the underlying capability improvements translate to real-world agent applications.

AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.