# NVIDIA explains automated workload recovery for distributed AI training failures

> Original title: Keep Distributed Training Running with Automated Workload Recovery

- Source: NVIDIA Technical Blog
- Published: 2026-11-02T22:43:05.000Z
- Bad Influence: https://badinfluence.news/items/aber4uvh4ntalto19vzop39jj
- Original: https://developer.nvidia.com/blog/keep-distributed-training-running-with-automated-workload-recovery/

## Summary

NVIDIA's Technical Blog describes how failures such as NCCL timeouts, unresponsive GPU nodes, and stragglers can disrupt distributed AI training. The post presents automated workload recovery as a way to keep training running after these disruptions.