SageMaker HyperPod splits LLM inference for better latency

Disaggregated Prefill and Decode (DPD) separates LLM inference phases onto dedicated GPU pools, aiming for consistent per-token latency.

What changed

• Introduced Disaggregated Prefill and Decode (DPD) for Amazon SageMaker HyperPod inference.
• DPD separates LLM prefill (compute-bound) and decode (memory-bound) phases onto distinct GPU pools.
• KV cache transfer between GPU pools uses Elastic Fabric Adapter (EFA) with GPU-Direct RDMA.
• An intelligent router directs long-context requests through the disaggregated path.

Why it matters

This addresses a common LLM inference bottleneck where prefill and decode contention on shared GPUs can lead to unpredictable latency, especially with long contexts. By dedicating separate GPU pools, one optimized for compute-bound prefill and another for memory-bandwidth-bound decode, DPD aims to deliver more consistent per-token latency. In practice: this means reduced stalls for concurrent requests and potentially higher throughput for workloads like chat assistants and RAG pipelines, especially when dealing with mixed prompt lengths. The ability to scale prefill and decode capacity independently is a significant operational advantage.

The catch

While DPD promises improved latency and throughput, the announcement does not specify the exact latency introduced by transferring the KV cache between GPU pools over EFA, nor the minimum number of GPUs required for each pool. The honest version: this feature adds complexity to the inference endpoint configuration, requiring explicit pdSpec configuration within InferenceEndpointConfig. Furthermore, while DPD is available in all SageMaker HyperPod regions, it necessitates EFA-capable instance types and the EKS orchestrator, limiting its immediate applicability to existing HyperPod deployments that do not meet these prerequisites.

Ship it

If you are running LLM inference on SageMaker HyperPod and experiencing inconsistent per-token latency under sustained concurrency, especially with long-context prompts, investigate enabling DPD. Ensure your cluster utilizes EFA-capable instances and the EKS orchestrator. You will need to update your InferenceEndpointConfig with the new pdSpec section. This feature pairs with SageMaker's existing intelligent routing and KV cache offloading capabilities for further optimization.

Bottom line: SageMaker HyperPod now offers disaggregated LLM inference to mitigate latency issues caused by prefill/decode contention.

— Filed to /aws