AWS explains detecting silent GPU data corruption on SageMaker HyperPod
Overview
AWS Machine Learning Blog explains how to detect silent data corruption (SDC) in PyTorch DistributedDataParallel training on Amazon SageMaker HyperPod, where a GPU returns wrong numbers without raising an error.
The post adds three checks to the training loop: non-finite gradient checks, a ring replay audit that recomputes micro-batches on another node, and weight fingerprint comparison.
The method requires a HyperPod cluster orchestrated by Amazon EKS with at least three GPU nodes, and a training job whose data pipeline can rebuild any rank's micro-batch for a given step, such as a seeded sampler. The post also shows how to map a suspect rank to a HyperPod node for containment and replacement, using kubectl and the sagemaker:StartClusterHealthCheck and sagemaker:BatchReplaceClusterNodes IAM permissions.
Written by AI from the articles below · updated Nov 11, 7:29 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- AWS Machine Learning BlogOfficialPickDetecting silent data corruption on Amazon SageMaker HyperPod
AIAWS explains how silent data corruption, where a GPU returns wrong numbers without raising an error, can be detected in PyTorch DistributedDataParallel training. The post adds three checks to the training loop: non-finite gradient checks, a ring replay audit that recomputes micro-batches on another node, and weight fingerprint comparison. It also shows how to map a suspect rank to a SageMaker HyperPod node for containment and replacement.
Heat Trend
- Current comparable heat
- 7
- Peak comparable heat
- 10Nov 11, 8:00 PM ET
- Change in past 24 hours
- –
The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.