AI Radar

Noteworthy AI conversations and writing, summarized from publisher notes.

SemiAnalysis on Sparse Attention, Memory Hierarchies, and GLM-5.3 Serving Economics

· SemiAnalysis

SemiAnalysis examines the practical memory and cost implications of sparse attention architectures like those in Z.ai's GLM-5 model series. Although top-k sparse attention reduces bandwidth and memory consumption during scaled dot-product attention, top-k selection still requires the full context to reside in HBM, meaning capacity bottlenecks persist. To address this, systems like SGLang's HiSparse implement hierarchical LRU caching and layer-wise overlapping to offload KV caches from HBM to host DRAM. Using InferenceX benchmarks for GLM-5.3, the analysis compares serving costs, throughput, and time to first token across NVIDIA GB200, GB300, and AMD MI355X under engines including Dynamo-SGLang, Dynamo-TRT-LLM, ATOM, and TileRT. The report also details the mathematics of DeepSeek Sparse Attention, covering the multi-head lightning indexer and Multi-Latent Attention roofline optimizations.

AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.