Ethan He on xAI's Video Agent Models and the Future of Multimodal AI
Ethan He, formerly of NVIDIA and Cosmos, joined xAI in early 2025 to help build video foundation models, contributing to the rapid development of Grok Imagine. He emphasized the critical role of strong compute infrastructure and iterative training cycles in accelerating video model development. Ethan detailed the challenges of training video models, including the need for synthetic paired data due to weak natural text-video alignment on the internet, and the importance of compressing video into latent tokens to manage sequence length. He highlighted that major improvements in video generation quality increasingly come from advances in language models that act as prompt rewriters or agents, rather than from video diffusion models themselves. Video agents combine language models with video generation and editing tools to produce longer, interactive, and real-time video content. Ethan envisions a future where generative UIs powered by video agents replace traditional interfaces, enabling highly interactive and personalized experiences. Ethan also discussed the technical challenges of real-time, long-horizon, interactive video world models, including context length management and temporal alignment of audio and video. He sees video agents as a stepping stone toward fully interactive world models that can be used in applications like robotics and immersive environments. Finally, Ethan shared his motivation for leaving xAI to focus more on language model research, believing that language models are the key bottleneck and driver for future multimodal AI advancements.
Original sources
Podcast Latent Space
Episode:Why Video Agent models are next — Ethan He, xAI Grok Imagine ↗
YouTube Latent Space
AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.