Read the original: PyTorch BlogOfficial· Josh Wills, Maximilian Böther, Ties Robroek and the Zephon team· Pick62/100AI score62/100
Zephon brings elastically deterministic data loading to TorchTitan
Original titleBringing Elastically Deterministic Data Loading to TorchTitan with Zephon
AISummary
Zephon is a data loader that tokenizes, packs, shuffles, and mixes data inside the training job and keeps global batches identical across GPU counts.
In the authors' tests, a standard loader shifted DCLM Core v1 scores by up to 0.82 points depending on GPU count, while Zephon's spread was 0.05 points. The TorchTitan integration lets a run resume on a different number of GPUs from the same checkpoint.
AIWhy it matters
The post shows how a data loader's GPU count changed results, and how Zephon keeps batches identical across GPU counts, which matters for fair ablation experiments.
Source: PyTorch Blog · pytorch.orgPublished · added here