Long Live the Short King: Why 4-hi HBM Wins
The article details how High Bandwidth Memory (HBM) has been crucial for AI, with chip designers packaging more HBM into accelerators. However, this trend is changing, with next-generation accelerators like Nvidia's Rubin Ultra moving to 8-hi stacks from 12-hi, reducing HBM per GPU. This shift is driven by both supply constraints and architectural reasons, as HBM wafers are scarce and cost is increasing. The report argues that for inference workloads, where memory bandwidth is key, 4-hi HBM offers the best cost-per-bandwidth, leading to lower cost per token. It explains that while capacity is important up to a point, beyond that, additional HBM capacity offers diminishing returns. The article concludes that hardware teams at frontier labs are increasingly seeing 4-hi HBM as the optimal configuration for many inference workloads, delivering better performance and total cost of ownership.
Original sources
Newsletter SemiAnalysis
AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.