Core Automation on Replacing Transformers for Continual Learning
Jerry Tworek and Rohan Anil, founders of new lab Core Automation, join Training Data to explain why they left frontier labs to pursue a transformer replacement. Tworek, formerly VP of Research at OpenAI where he led the o1/strawberry reasoning effort, describes a personal reckoning: after scaling reinforcement learning to unprecedented levels, benchmark scores rose but real-world task performance did not, because training distributions fail to match messy deployment distributions. Anil, ex-pre-training lead on Gemini and longtime Google Brain/Anthropic researcher, adds that current transformer compute spending is inefficient, pointing to shallow networks and one-token-at-a-time autoregressive decoding as structural bottlenecks. The pair argue that transformers are fundamentally incapable of continual learning. In-context learning hits a practical wall around 20 minutes of usage, and fine-tuning suffers from catastrophic forgetting and low data efficiency. They believe a new architecture is required to enable learning at test time on user data, and that the biggest labs are too locked into profitable transformer scaling cycles, and competitors too busy cloning coding agents, to explore alternatives. Core Automation is structured as the most automated research lab possible, aiming to run one architecture experiment per day by combining coding agents, automated kernel generation, and integrated teams spanning research through GPU kernel work. On efficiency, Anil argues that pre-training and RL must be optimized end-to-end rather than as separate stages, and that better optimizers (the Shampoo/Soap lineage he helped develop for Gemini 1.5 Flash) co-evolve with architecture, opening space for harder-to-train models. The team also emphasizes custom kernels as a critical enabler, citing a QR decomposition competition where human-plus-coding-agent produced a kernel 60x faster than cuSolver, a gap current frontier models cannot close. On AGI, Tworek defines it as a model that improves itself without a human in the loop, which he says the current path does not deliver.
Original sources
Podcast Training Data
Episode:Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil ↗
YouTube Sequoia Capital
Video:Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil ↗
AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.