Google launches native PyTorch backends for vLLM and SGLang on TPU

Google has announced native PyTorch backends for the vLLM and SGLang inference engines, enabling developers to run serving workloads using standard PyTorch tools on TPU hardware.confirmed The launch, detailed in a post on the Google Developers blog, marks a shift from previous JAX-based implementations to a native PyTorch experience powered by TorchTPU.

The evidence

The new engines were built in collaboration with open-source project maintainers, specifically Inferact for vLLM and RadixArk for SGLang, according to the Google Developers blog post.reported These implementations allow tensors to be managed as standard PyTorch tensors on a TPU device without wrapper types, running eagerly or compiling via torch.compile using XLA. Collectives operate through torch.distributed, making the TPU a native PyTorch device.

Each device runs its own PyTorch process, preserving each framework’s worker model, scheduler, KV cache manager and API server while utilising TPU-specific implementations underneath, the blog post states. Most models use standard upstream PyTorch model definitions, with TPU-specific optimisations applied in kernels and layer implementations. Custom kernels written in Pallas are called directly from PyTorch as custom operators inside the compiled graph.

The decision

Both engines are designed for the full range of production workloads, including agentic, multi-turn, interactive, low-latency, high-throughput batch, long-context, multimodal and very large model scenarios, according to the announcement.reported Features include automatic prefix caching, KV cache offloading to host memory and Managed Lustre through TPU Sync, and tool calling for agentic workloads. Interactive use cases benefit from speculative decoding with MTP, EAGLE-3, DFlash, DSpark and n-gram drafters.

High-throughput batch processing supports data-parallel attention with expert parallelism, while long-context tasks utilise context parallelism for prefill and decode, FP8 KV cache and host KV offloading.reported Disaggregated serving allows prefill and decode to run on separate hosts over TPU Sync, including between different parallel layouts. Very large models can leverage hybrid and sparse-attention architectures with various parallelism types and precision formats such as FP8, FP4 and INT8.

What to make of it

Agentic workloads demand several features simultaneously, as agents repeatedly resend expanding multi-turn conversations, requiring efficient caching to avoid recomputing prefixes, the blog post explains. Both engines keep reusable prefixes in HBM and automatically move older prefixes to host memory as a second tier when host offload is enabled. Two features move the KV cache away from the computing device: disaggregated serving and tiered KV offload.

Disaggregated serving runs prefill and decode on separate TPU hosts, requiring the KV cache to travel between them, while tiered KV offload moves parts of the cache to host DRAM and Managed Lustre when they no longer fit in HBM. Both engines use TPU Sync, an open-source C++ library for KV cache transfer and management on TPU, to handle these transfers, which must overlap computation to avoid adding directly to latency.

In practice

Startups can now deploy vLLM and SGLang on TPU using native PyTorch, simplifying debugging and customisation for teams already familiar with PyTorch, according to the Google Developers blog.reported Access to these engines is available immediately through the announced backends, though availability may depend on hardware access to Ironwood TPU or other TPU generations.confirmed Limitations include the need to manage KV cache transfers carefully to avoid latency penalties, as disaggregated serving and host offload require overlapping computation with data movement.

The engines support a wide range of features, including automatic prefix caching, speculative decoding and disaggregated serving, but users must configure these appropriately for their specific workloads. Public benchmarks demonstrate state-of-the-art performance on Ironwood TPU, but results may vary depending on model size, context length and specific workload characteristics. No fix has been confirmed for any potential issues, as this is a new release.