Exact sequence parallelism for packed multimodal masks.
It picks ring attention, a sparse all-to-all or Ulysses per layer, matches single-GPU numerics, and beats a FlashMask + Ulysses hybrid in 47 of 48 cases on up to 128 H100s.
Luma details exact sequence parallelism for packed multimodal masks
AISummary
Luma says its sequence parallelism method selects ring attention or a sparse all-to-all per layer and matches single-GPU numerics. The company reports it beat a FlashMask plus Ulysses hybrid in 47 of 48 cases on up to 128 H100 GPUs.
Post on XView on X
Source: Luma · x.comPublished · added here