OpenAI Jalapeño: Better Than Nvidia Blackwell
OpenAI has developed an inference chip named "Jalapeño," which was recently announced at Hot Chips. This chip, built in partnership with Broadcom, is designed exclusively for LLM inference and achieved a manufacturing tape-out in approximately 16 months. Despite being a first-generation chip, Jalapeño reportedly leads the industry, outperforming Nvidia, AMD, and Google chips on multiple top open-source models, a feat attributed to extreme hardware-software co-design. Contrary to some beliefs, Jalapeño is a generalized inference chip, not specialized for OpenAI models, and delivers high performance across various workloads. The chip's design prioritizes power efficiency, achieving superior token throughput per MW compared to competitors like Nvidia's Blackwell and Vera Rubin, even without speculative decoding. OpenAI's architectural choice of a homogenous chip pool, rather than disaggregating prefill and decode, aims for adaptability to changing workload mixes over time. The A0 stepping of Jalapeño shows impressive results, with a B0 stepping already in the fab promising further improvements in performance per watt.
Original sources
Newsletter SemiAnalysis
AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.