Amazon Web Services announced Amazon SageMaker HyperPod Inference Gateway on September 18, 2026, a Kubernetes-native, GPU-aware routing system for large language model inference that deploys as a single managed add-on for Amazon EKS on existing HyperPod infrastructure. AWS said the gateway can reduce first-token latency by up to 82%.
The Routing Problem Behind the Gateway
According to AWS, default Kubernetes load-balancing algorithms such as round-robin and least-connections have no visibility into GPU state: which pods have saturated KV caches, which are mid-way through long-context generations, and which already have the LoRA adapter a request needs loaded in memory. The company said requests pile up behind busy pods while idle capacity remains unused, first-token latency spikes above four seconds during traffic bursts, utilization becomes uneven and unpredictable, and operators over-provision to compensate. AWS described a scenario in which a chatbot user waiting 4.4 seconds for a first token instead sees it in under 800 milliseconds.
Two-Tier Architecture
The gateway uses a two-tier design built on Kubernetes-native primitives. AWS said it uses real-time GPU signals to place every inference request on the best-suited pod. Tier 1 installs directly on each HyperPod or EKS cluster as the amazon-sagemaker-hyperpod-inference add-on and consists of three components, all built on the open-source Gateway API Inference Extension. Envoy Gateway, a layer-7 proxy, terminates incoming HTTPS traffic and exposes a single private endpoint per cluster. The Body-Based Router inspects each incoming OpenAI-compatible request body, extracts the model field, and routes the request to the correct model pool, so one gateway can serve multiple models.
The Endpoint Picker consumes real-time Prometheus metrics from every model-serving pod and applies a weighted scoring algorithm across scorers covering KV cache utilization, queue depth, LoRA adapter residency, prefix cache hit rate, and running requests. Each scorer carries a configurable weight, allowing routing behavior to be tuned for a specific workload, such as latency-sensitive chat versus throughput-optimized batch.
Tier 2, the Global Inference Router, is listed as coming soon. AWS said it will add fleet-wide coordination across multiple clusters and regions, with cross-cluster failover, global rate limiting, and cost-aware traffic shaping. Tier 2 builds on top of Tier 1, while each cluster’s per-cluster gateway continues to handle local routing.
Deployment, Failure Handling, and Observability
Deployment consists of a single aws eks create-addon command and one declarative InferenceGatewayConfig custom resource that defines models and routing behavior, with existing model server deployments discovered through pod labels. AWS said the installation requires no sidecars, no service mesh, and no application code changes. The gateway exposes a standard OpenAI-compatible endpoint over HTTP; according to AWS, existing client code works unchanged, with no SDK changes and no SigV4 signing for inference traffic.
For workloads serving fine-tuned LoRA adapters on a shared base model, the Endpoint Picker’s LoRA Affinity Scorer routes adapter requests to pods that already have the requested adapter resident in GPU memory; if no pod has it loaded, the request goes to the pod with the most available capacity. AWS said this eliminates adapter swap latency.
Documented failure behaviors cover pod failure, pool exhaustion, cluster failure, and regional failure. On pod failure, the Endpoint Picker excludes pods with stale metrics and routes to healthy pods, recovering automatically when metrics resume. On pool exhaustion, the gateway returns HTTP 429 with a Retry-After header while autoscaling adds capacity. On cluster failure, the Global Inference Router detects a stale heartbeat and redirects traffic within 35 seconds, with gradual ramp-up when the cluster is reintroduced. On regional failure, cross-region routing activates automatically, which AWS said carries higher latency but no availability impact.
The gateway emits metrics at the pod, pool, cluster, and fleet levels: KV cache utilization, queue depth, running requests, and adapter residency through Prometheus at the pod level; request totals, duration histograms, and token counts through Prometheus and Grafana at the pool level; average KV cache, error rate, and P99 latency through Amazon CloudWatch at the cluster level; and routing decisions, failover events, and rate limit hits through CloudWatch at the fleet level.
AWS-Reported Benchmark Results
AWS said it benchmarked four models ranging from 8B to 235B parameters on p5.48xlarge instances with H100 GPUs and g5 instances with A10G GPUs. All traffic was routed through internal Application Load Balancers, matching the path a production request travels, with a dedicated client node group generating controlled load and model servers isolated on a separate server node group. Every result uses the gateway’s default routing configuration with no tuning and is measured against a Kubernetes round-robin baseline on the same model replicas, according to AWS.
In the reported results, a mixed-generation GPU fleet cut time-to-first-token P95 and P99 latency by 97% each for Llama-3.1-8B, with an 8% throughput increase, and by 98% and 97% for Qwen3-32B, with a 50% throughput increase. Under bursty traffic, Llama-3.1-70B posted P95 and P99 reductions of 94% and 98% with 12% higher throughput, while Qwen3-235B showed comparable P95 latency and an 89% lower P99. With shared prompt prefixes, Llama-3.1-8B P95 and P99 latency fell 26% and 43%.
AWS said that on a fully uniform fleet under steady traffic the gateway performs on par with round-robin, and it defined comparable results as differences within run-to-run variance. The company said the improvements are largest where round-robin struggles most: mixed hardware, bursty demand, and shared prompt prefixes.
Availability and Roadmap
AWS describes the gateway as conformant with the Kubernetes Gateway API and its Inference Extension, configured through a single custom resource definition, and compatible with any OpenAI-compatible model server, including vLLM, SGLang, and TGI. Management works through kubectl, GitOps, Helm, and ArgoCD, with installation, upgrades, and rollback handled through the EKS add-on lifecycle.
Tier 1 per-cluster routing is available as of September 18, 2026, in regions where the inference add-on is available. Beyond the Global Inference Router, AWS’s named roadmap items include canary traffic splitting, which will route a percentage of traffic to new model versions using InferenceModelRewrite custom resources, and flow control that classifies requests as Critical, Standard, or Sheddable with per-band admission control.
Credit: Source link
