Luma proposes exact sequence parallelism for packed multimodal attention masks

Luma says it has developed an exact sequence parallelism method for packed multimodal attention masks.

The method chooses between ring attention and a sparse all-to-all for each layer and, according to Luma, matches single-GPU numerics. Luma reports that it beat a FlashMask plus Ulysses hybrid in 47 of 48 cases on up to 128 H100 GPUs. The claims come from a post by Luma on X, and no independent evaluation is cited.