SemiAnalysis on Computation and Data Movement for MoE Inference
SemiAnalysis examines how Mixture of Experts (MoE) models alter the structural and economic demands of frontier AI inference serving. The analysis distinguishes four operating regimes with differing arithmetic intensities and network requirements: prefill, midfill, decode attention, and decode experts. Rather than treating these regimes as a uniform workload on aggregated servers, modern inference architectures increasingly disaggregate stages across specialized worker pools coordinated by cluster schedulers. The piece details how multi-turn agentic workflows repeatedly cycle through context extension and generation loops, driving large KV-cache state transfers between workers. To maintain high throughput and low latency, inference clusters organize reusable context into immutable blobs managed across scale-out memory pools and high-bandwidth datacenter networks.
Original sources
Newsletter SemiAnalysis
AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.