AWS SageMaker Now Splits One GPU Into Seven

Amazon Web Services on Friday, Nov. 6, added GPU partitioning to Amazon SageMaker AI Inference, letting customers split one NVIDIA GPU into as many as seven hardware-isolated slices and run a different model in each. The feature, announced on the AWS Machine Learning Blog, brings NVIDIA Multi-Instance GPU (MIG) to managed real-time endpoints without customers running their own Kubernetes stack.

Key takeaways

• SageMaker endpoints can now divide each GPU into up to seven MIG partitions, each with its own compute and memory.
• In AWS tests, half of an H100 beat a whole A100-40GB on throughput and cost 18% to 34% less per million output tokens at higher loads.
• Customers still pay for the full GPU instance; partitioning only raises how much of it they can use.

How SageMaker GPU partitioning works

Until now, the smallest GPU unit a SageMaker endpoint could request was one whole accelerator. An ml.p4d.24xlarge with eight A100s could host only eight models, whether each needed 40 GB or 14 GB, AWS wrote.

The new feature adds two optional fields to existing API calls. AcceleratorPartitionLayouts, set on an endpoint configuration, declares how every GPU is carved. AcceleratorPartitionType, set on an inference component, picks the slice a model runs on. AWS says there are no new API operations, resource types or IAM actions, and invocation does not change.

In AWS's example, splitting each A100 on a p4d instance into 4g, 2g and 1g slices yields 24 partitions on one machine. Carving added roughly 17 seconds to endpoint provisioning in AWS testing.

Supported SageMaker instances and Regions

Partitioning works on four instance families: P4d (NVIDIA A100, 40 GB, up to seven partitions), P5 (H100, 80 GB, seven), G7 (RTX PRO 4500 Blackwell, 32 GB, two) and G7e (RTX PRO 6000 Blackwell Server Edition, 96 GB, four). It is offered in seven Regions, including US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo) and Europe (Frankfurt).

Salesforce helped define requirements and validated the first version. "Many inference workloads do not require an entire GPU," said Richard Chang, a software engineering architect at Salesforce, in a statement in the AWS post.

H100 slice vs. full A100: AWS benchmark

AWS compared one mig-3g.40gb partition of an H100 with one whole A100-40GB, both serving Llama-3.1-8B-Instruct across five traffic shapes and concurrency from 2 to 512. The partition delivered 53% to 89% more output tokens per second at each scenario's best cost point. Time to first token fell 5.6% to 47.5%, and inter-token latency fell 8.9% to 42.1%.

The cost edge appeared only above a crossover: concurrency 64 for 100-token prompts, and 4 for 8,000-token prompts. Below that, the whole A100 was cheaper. On 6,000- and 8,000-token workloads at peak load, the A100 failed a third to half of requests, while the partition failed none. AWS based costs on Oregon on-demand prices of about $25.25 an hour for ml.p4d.24xlarge and $63.30 for ml.p5.48xlarge. These are vendor-run tests.

Context: limits to know

A layout is fixed when an endpoint is created; changing it means a new endpoint and a traffic migration. Single-model endpoints, serverless endpoints, heterogeneous instance pools and asynchronous inference are not supported. According to the SageMaker developer guide, partitioned endpoints do not publish GPU, CPU or memory utilization metrics to CloudWatch. Isolation also covers only the GPU; models still share host CPU, memory, storage and networking.

The release follows other recent AWS AI tooling, such as the open-source Strands harness.

What's next

AWS frames the change as giving every GPU "several sizes rather than one." The feature suits stable fleets of small and mid-size models, such as fine-tuned variants, embedders, rerankers and guardrail classifiers. Teams that reshuffle models daily, or need more than one GPU per model, gain less. AWS published a sample notebook on GitHub for testing.

FAQ

Does GPU partitioning lower my SageMaker bill?
Not directly. AWS bills the full instance as before. Savings come from packing more models onto the same hardware or using a slice of a newer GPU.

Can I change partition sizes on a running endpoint?
No. AWS says an update with a different layout returns a 400 error. You must create a new endpoint.

Which GPUs support the most partitions?
A100 and H100 instances support up to seven per GPU. G7e supports four and G7 two.

References

• SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints — AWS Machine Learning Blog (Nov. 6, 2026)
• Using GPU partitions on SageMaker AI real-time endpoints — Amazon SageMaker AI Developer Guide