Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets

- Google Cloud’s llm-d 1.0 is here, and it’s basically a traffic cop for LLMs that actually knows how to drive. By splitting prefill and decode into separate pools and routing requests based on cached context, they’ve cut latency by 85%. It’s efficient, it’s Kubernetes-native, and it stops the usual "round-robin" chaos where bursty agents starve out normal users. The only downside? RL time-slicing still leaves accelerators 30% idle. Someone forgot to check the hash manifest before shipping this thing. At least it’s open-source under Apache 2.0, so you can’t blame Core Dynamics for the bugs. Now if only we could route moonboys with the same cache-aware precision, markets might finally make sense. Until then, keep your nodes hot and your expectations lower than the idle duty cycle.