SemiAnalysis on Sparse Attention, Memory Hierarchies, and GLM-5.3 Serving Economics
SemiAnalysis examines the practical memory and cost implications of sparse attention architectures like those in Z.ai's GLM-5 model series. Although top-k sparse attention reduces bandwidth and memory consumption during scaled dot-product attention, top-k selection still requires the full context to reside in HBM, meaning capacity bottlenecks persist. To address this, systems like SGLang's HiSparse implement hierarchical LRU caching and layer-wise overlapping to offload KV caches from HBM to host DRAM. Using InferenceX benchmarks for GLM-5.3, the analysis compares serving costs, throughput, and time to first token across NVIDIA GB200, GB300, and AMD MI355X under engines including Dynamo-SGLang, Dynamo-TRT-LLM, ATOM, and TileRT. The report also details the mathematics of DeepSeek Sparse Attention, covering the multi-head lightning indexer and Multi-Latent Attention roofline optimizations.
Original sources
Newsletter SemiAnalysis
Issue:How GLM5.3 Sparse Attention Affects HBM Memory Usage ↗
AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.