Where Does a Robot Think – On-Device vs Datacenter Inference
The article explores the emerging frontier of physical AI, specifically in robotics, contrasting it with traditional LLMs. It emphasizes that unlike LLMs where hardware adapts to the model, robotics requires models to fit hardware constraints due to strict latency requirements and high unit costs. Frontier robot models are currently much smaller than LLMs, typically ranging from 5 to 14 billion parameters. The field is divided on whether to run inference on-board the robot or offload it to datacenters, with larger builders like NVIDIA and Google DeepMind leaning towards cloud-based heavy cognition. The article details how hierarchical robot models, with separate planning and action layers, allow the compute-intensive planning layer to be moved to datacenters, while the fast action layer remains local. It also explains why frontier robot models are FLOPs-hungry and how edge silicon struggles to keep up, necessitating off-device processing for generalist robots. Optimizing local compute for safety and high-frequency control loops, alongside robust network interfacing, becomes crucial when shipping robot models off-device.
Original sources
Newsletter SemiAnalysis
Issue:Where Does a Robot Think – On-Device vs Datacenter Inference ↗
AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.