TRL's AsyncGRPO adds tree packing, cutting agentic RL trainer compute 26x

Hugging Face's TRL team has shipped tree packing in AsyncGRPO, according to Lewis Tunstall.

Forked multi-turn rollouts share history, and tree packing forwards each unique token once instead of re-forwarding the shared history. On a real agentic RL run on MiMo Envs, Tunstall says it reached the same reward with 26x less trainer compute.

Amine of the TRL team says the first version used FlexAttention, but the shipped version is a single FA3 varlen call. He says that when nothing can be packed, the path costs about 1%.