AWS Adds NVIDIA GPU Partitioning to SageMaker Inference Endpoints – Unite.AI

Amazon Web Services (AWS) announced NVIDIA Multi-Instance GPU support for Amazon SageMaker Inference on November 6, 2026, letting multiple hardware-isolated models share one accelerator on real-time endpoints instead of assigning each deployment an entire GPU.

The feature addresses two constraints of whole-GPU allocation. Accelerator count caps how many models an instance can host, regardless of how little memory or compute each model uses, and customers considering a newer GPU must allocate the entire device. Multi-Instance GPU (MIG) instead gives each partition a fixed share of compute units and a dedicated block of memory, with isolation enforced in hardware. Because each model remains resident, the arrangement also avoids the delay incurred when a multi-model endpoint loads a model onto its shared device for first use.

Partitioning is limited to real-time endpoints that use inference components, SageMaker’s construct for deploying several models to one endpoint and setting the compute each receives. Single-model endpoints, serverless endpoints and heterogeneous instance pools cannot be partitioned. Asynchronous workloads are also outside the feature’s scope because asynchronous invocation cannot address an inference component.

Salesforce helped define the requirements and validate the initial implementation. Richard Chang, a Salesforce software engineering architect, said the company expected MIG to consolidate workloads safely, improve GPU utilization and reduce inference costs while preserving predictable performance and isolation. Salesforce is preparing to validate the software with representative production workloads, so those benefits are expectations rather than reported production results.

Two API Fields Control Partitioning

The SageMaker AI developer documentation defines two optional fields: AcceleratorPartitionLayouts declares the carve on a production variant, and AcceleratorPartitionType selects the profile used by an inference component. There are no new API operations, resource types or AWS Identity and Access Management actions, and omitting both fields preserves existing endpoint behavior.

The selected layout applies uniformly to every accelerator on every instance in the variant. AWS’s example divides each of eight A100s on an ml.p4d.24xlarge into mig-4g.20gb, mig-2g.10gb and mig-1g.5gb partitions, producing 24 partitions across the endpoint. Each inference-component copy occupies one requested partition, while InvokeEndpoint continues to address models by component name without a partition identifier. In its testing, AWS said carving the accelerators added roughly 17 seconds to provisioning.

Isolation covers GPU compute and memory, not the entire host. Models on one instance still share host CPU, host memory, storage, networking and endpoint routing. A component that requests a profile absent from the layout is rejected rather than given a whole GPU. Changing a running endpoint’s layout requires a new endpoint and traffic migration; moving a component to another profile requires a new component.

AWS Tests an H100 Partition Against a Full A100

AWS compared one mig-3g.40gb H100 partition with a whole A100-40GB, using Llama-3.1-8B-Instruct on production SageMaker real-time endpoints. Both sides used one model replica, identical containers, tokenizers and client configurations, with genai-bench driving five traffic shapes at concurrency levels from 2 to 512. NVIDIA’s profile guide allocates the tested H100 profile three of seven streaming-multiprocessor shares and four of eight memory units. The two configurations have comparable memory capacity but different compute allocations.

Across the tests, AWS reported 53% to 89% more output tokens per second from the H100 partition at the load where each scenario’s cost benefit was largest. Inter-token latency was 8.9% to 42.1% lower, and time to first token was 5.6% to 47.5% lower. At the highest concurrency on 6,000-token and 8,000-token workloads, the A100 missed the serving deadline on one-third to one-half of requests while the partition failed none.

AWS’s cost analysis, presented as a case-by-case comparison rather than a universal verdict, found an 18% to 34% lower cost per million output tokens once traffic passed a workload-specific break-even point. That point occurred at concurrency 64 for 100-token prompts and concurrency 4 for 8,000-token prompts. Below it, the whole A100 remained cheaper, leaving short-input or low-concurrency workloads potentially better suited to the older whole GPU. The unit-cost comparison divided on-demand instance prices by the number of tested units on each instance.

Supported Hardware and Operating Limits

The launch covers four instance families, with partition counts and profiles determined by the accelerator:

GPU partitioning is available in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt) and Europe (Ireland), subject to regional instance availability and endpoint quota. No partition spans accelerators, so models requiring a full GPU or multiple GPUs still need whole devices.

Customers continue paying for the full GPU instance; partitioning can increase usable capacity but does not reduce its hourly price. Partitioned endpoints and their inference components also do not publish GPU, CPU or memory utilization metrics to CloudWatch, although invocation and latency metrics remain available. On P4d, the mig-1g.10gb profile requires NVIDIA’s R525 driver or newer, and AWS directs customers to a newer inference AMI because the default image does not provide it.

For capacity planning, the launch turns each supported accelerator into several fixed GPU sizes, but denser hardware-isolated deployment comes with full-instance billing, layouts that cannot be altered in place and reduced utilization telemetry. AWS’s benchmark shows the value of measuring actual traffic: the H100 partition delivered performance and cost advantages under heavier long-context conditions in those tests, while the break-even results left a whole A100 cheaper for lighter workloads.