AWS inference gateway cuts first-token latency by up to 82%

AWS has announced Amazon SageMaker HyperPod Inference Gateway, a Kubernetes-native routing system for teams serving large language models (LLMs) on GPU clusters. It installs as a single EKS managed addon on existing HyperPod infrastructure and uses real-time GPU signals to place each inference request on the best-suited pod. AWS reports first-token latency reductions of up to 82%, with one example moving a chatbot user from 4.4 seconds to under 800 ms.

The problem AWS describes: default Kubernetes load balancers have no view inside the GPU. Round-robin and least-connections algorithms cannot tell which pods have saturated KV caches, which are mid-way through long-context generation, or which already hold the requested LoRA adapter in memory. Requests queue behind busy pods while idle capacity sits unused.

Two tiers, no code changes

Tier one installs on each cluster as the amazon-sagemaker-hyperpod-inference addon, built on the open-source Gateway API Inference Extension. It combines an L7 proxy that terminates HTTPS traffic behind one private endpoint per cluster, a body-based router that reads the model field from OpenAI-compatible requests and forwards to the right model pool, and the EPP, which consumes Prometheus metrics from every serving pod and picks a backend through weighted scoring. Scorer weights are configurable, so latency-sensitive chat and throughput-oriented batch work can be tuned separately. Tier two adds cross-cluster failover, global rate limiting and cost-aware traffic shaping across clusters and regions.

Deployment requires a label on existing model server deployments and one declarative InferenceGatewayConfig resource. No sidecars, no service mesh, no application changes; clients keep calling a standard OpenAI-compatible endpoint over plain HTTP. A LoRA Affinity Scorer routes adapter requests to pods that already have the adapter resident, avoiding swap latency.

AWS benchmarked four models from 8B to 235B parameters on p5.48xlarge (H100) and g5 (A10G) instances, routing all traffic through internal Application Load Balancers. The company says gains were largest with mixed GPU generations, bursty demand and shared prompt prefixes, and that the further a fleet drifts from uniform, the bigger the benefit.

Configuration details are in the AWS announcement.