AI Radar

Noteworthy AI conversations and writing, summarized from publisher notes.

SemiAnalysis: TPU Inference Externalization Full Steam Ahead - InferenceX

· SemiAnalysis

SemiAnalysis provides an in-depth analysis of Google's TPUv7 Ironwood, highlighting its performance-per-dollar advantage in inference workloads against NVIDIA's B200 and B300. The report indicates Ironwood delivers up to 50% better performance per dollar, with a 19% lower cost than B200 and 34% lower than B300 at 100 tokens/s/user interactivity. It also discusses the new TorchTPU stack, which allows TPUs to function as native PyTorch devices, a significant improvement over the previous TorchAX approach. Google's strategy to externalize TPUs for inference is examined, including plans to support open-weight models like Qwen3.5 397B, Kimi K3, and GLM5.3. The article also touches on future optimizations like disaggregated serving and speculative decoding, and the expected release of the TorchTPU stack from private beta around October.

AI Radar wrote this summary from the publisher’s notes with AI, not from a transcript. How it works.