Google describes Pathways single-controller training for large models
Overview
Google's developer blog explains Pathways, the distributed system it built to train its own large models including Gemini, and announces its General Availability on Google Cloud Platform.
In the post, a single client describes the whole training job as a graph, and the distributed runtime places that work, moves tensors between devices, and keeps accelerators busy.
The post contrasts this with SPMD execution, in which lockstep collectives make pipeline stages and reinforcement learning loops harder to overlap, and a single machine failure restarts the run, according to Google's description.
Written by AI from the articles below · updated Nov 9, 7:36 AM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- Google Developers BlogOfficialPickGoogle explains how Pathways coordinates large-model training with a single client
AIGoogle's developer blog explains Pathways, the distributed system it built to train its own large models including Gemini, and announces its General Availability on Google Cloud Platform. The post describes how one client declares a training job as a graph, while the distributed runtime places the work, moves tensors between devices, and keeps accelerators busy. It contrasts this with SPMD execution, where lockstep collectives make pipeline stages and RL loops harder to overlap and a single machine failure restarts the run.
Heat Trend
Not enough continuous observations to show a trend yet.