Amazon SageMaker Inference adds GPU partitioning with Multi-Instance GPU
AIAmazon SageMaker Inference now supports GPU partitioning, using NVIDIA Multi-Instance GPU to divide one physical GPU into isolated partitions that each host a model. The feature is available on P4d, P5, G7, and G7e instance families in seven AWS Regions, and customers are billed for the full GPU instance.
AIWhy it matters.
The post gives per-GPU-type partition profiles and a benchmark against a whole A100, showing when a fractional GPU beats a full one.
Source details
AWS Machine Learning Blog · Shantanu Tripathi · aws.amazon.comPublished · added here
Original title: SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints