Comparing the time per MoE layer between Prime Mega MoE and our prime-rl baseline:
- The forward pass is up to 3.11× faster
- The backward pass is up to 1.87× faster
- The full layer with activation checkpointing is up to 2.47× faster
Prime Intellect's Mega MoE speeds up MoE layers up to 2.47×
AIPrime Intellect says its Prime Mega MoE runs the forward pass up to 3.11× faster and the backward pass up to 1.87× faster than its prime-rl baseline per MoE layer. With activation checkpointing, the full layer is up to 2.47× faster.
@PrimeIntellect
Part of a thread · earlier post
Source details
Prime Intellect · x.comPublished · added here
Original title: Comparing the time per MoE layer between Prime Mega MoE and our prime-rl baseline: