Google adds long-context training support to MaxText on Cloud TPUs
AIGoogle Cloud announces long-context training support in MaxText, its open-source JAX training framework, with four context-parallel strategies on Cloud TPU. Google says Llama 3.1 8B reaches 70% Model FLOPs Utilization on TPU 5p at sequence lengths up to 4 million tokens, and context can extend to 14 million tokens with TPU 7 maintaining 50% MFU below 6 million tokens. The feature is available in MaxText v0.2.4.
Source details
Google Developers Blog · developers.googleblog.comPublished · added here
Original title: Announcing Long-Context Training on TPUs with MaxText Context Parallelism