Amazon SageMaker Inference adds GPU partitioning with NVIDIA MIG
Overview
Amazon SageMaker Inference now supports GPU partitioning, using NVIDIA Multi-Instance GPU (MIG) to split one physical GPU into isolated partitions that each host a model.
Customers are billed for the full GPU instance, as before. The feature is available on the P4d, P5, G7, and G7e instance families in seven Regions: US East (N. Virginia and Ohio), US West (Oregon), Asia Pacific (Singapore and Tokyo), and Europe (Frankfurt and Ireland).
Single-model endpoints, serverless endpoints, and heterogeneous instance pools cannot be partitioned.
An AWS Machine Learning Blog post reports a benchmark in which one mig-3g.40gb partition of an H100 delivered 53 to 89 percent more output tokens per second than a whole A100-40GB at high concurrency. The post says the partition was also cheaper per million output tokens above a traffic-dependent crossover point.
Written by AI from the articles below · updated Nov 6, 1:31 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- AWS Machine Learning BlogOfficialPickAmazon SageMaker Inference adds GPU partitioning with Multi-Instance GPU
AIAmazon SageMaker Inference now supports GPU partitioning, using NVIDIA Multi-Instance GPU to divide one physical GPU into isolated partitions that each host a model. The feature is available on P4d, P5, G7, and G7e instance families in seven AWS Regions, and customers are billed for the full GPU instance. In the blog's benchmark, one mig-3g.40gb partition of an H100 delivered 53 to 89 percent more output tokens per second than a whole A100-40GB at high concurrency, and was cheaper per million output tokens above a traffic-dependent crossover point.
Heat Trend
Not enough continuous observations to show a trend yet.