Zephon adds elastically deterministic data loading to TorchTitan
Overview
The Zephon team introduces Zephon, a data loader that tokenizes, packs, shuffles and mixes data inside the training job and keeps global batches identical regardless of GPU count.
Its TorchTitan integration replaces TorchTitan's default data loader with a Zephon Pipeline built from a user's data recipe, with the rest of the training run unchanged, according to a PyTorch Blog post.
In the authors' tests, a standard loader shifted DCLM Core v1 scores by up to 0.82 points depending on GPU count, while Zephon's spread was 0.05 points. The integration lets a run resume from the same checkpoint on a different number of GPUs.
Written by AI from the articles below · updated Oct 16, 1:27 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- PyTorch BlogOfficialPickZephon brings elastically deterministic data loading to TorchTitan
AIZephon is a data loader that tokenizes, packs, shuffles, and mixes data inside the training job and keeps global batches identical across GPU counts. In the authors' tests, a standard loader shifted DCLM Core v1 scores by up to 0.82 points depending on GPU count, while Zephon's spread was 0.05 points. The TorchTitan integration lets a run resume on a different number of GPUs from the same checkpoint.
Heat trend
Not enough continuous observations to show a trend yet.