This is an open-source tool for training large language models across multiple GPUs.
It's pretty efficient:
It reuses memory and transfers model data while your GPUs run other calculations, making training much faster.
Yandex published an example where training Llama 3 70B on 256 A100 GPUs took about 26% less time than with a PyTorch FSDP setup.
Our technology helps train LLMs up to 26% faster — and it's open source. Training a model takes a lot of computing power, but coordinating the work across multiple GPUs can leave up to 30% of their capacity underutilized. GPUs spend time waiting for one another instead of training. This is why we developed YaFSDP and subsequently released it as open source.
Santiago · x.comPublished · added here
Original title: This is an open-source tool for training large language models across multiple GPUs.
