Engrams Embedding Entendre: Codesign for Efficient DRAM/SSD Offloading
The article details Engram, a model architecture optimization that uses learned multi-token lookups to reduce the need for HBM capacity by retrieving vectors directly. This design is naturally codesigned for parameter offloading, allowing Engram tables to reside outside HBM in host DRAM or even SSDs. The analysis includes performance benchmarks of DeepSeek-V4.1-Flash with Engram offloading on various NVIDIA GPU SKUs and AMD's MI355X, showing NVIDIA's significant performance and cost-per-dollar advantage. It also examines the benefits of offloading Engram to DRAM, which can improve performance and reduce communication overhead by enabling lower tensor parallelism. While SSD offloading is explored, the benchmarks suggest it is not yet viable for production serving due to performance and cost trade-offs. The report also discusses the impact of removing Engram on model performance and the interplay between memory features and expert selection.
Original sources
Newsletter SemiAnalysis
Issue:Engrams Embedding Entendre: Codesign for Efficient DRAM/SSD Offloading ↗
AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.