Skip to content
Trending storyDeveloping

Prime Intellect extends DeepSeek's Mega MoE kernel with a fused backward pass

6 articles2 sourcessince Oct 29Last article 1h ago ·

Overview

AISummary of 6 articles

Prime Intellect releases Prime Mega MoE, which adds a fused, overlapping backward kernel to DeepSeek's Mega MoE, a kernel that fuses MoE routing, dispatch, compute and return.

DeepSeek's version covers only the forward pass. Prime Intellect says the kernel runs the forward pass up to 3.11× faster and the backward pass up to 1.87× faster per MoE layer than its prime-rl baseline, and up to 2.47× faster for the full layer with activation checkpointing.

At the training-step level, the company reports steps 34% faster on Qwen3-30B-A3B and 32% faster on Qwen3-235B-A22B than the prime-rl baseline, and 48% and 55% faster than TorchTitan on the same models. MarkTechPost reports the same figures and says the kernel supports BF16, MXFP8 on Blackwell, and blockwise FP8 on Hopper, shipping in prime-rl under Apache 2.0. All benchmark figures are Prime Intellect's own.

Written by AI from the articles below · updated Oct 29, 3:27 PM ET

Check the sources:

Latest developmentOct 29, 2:28 PM ET
Prime Intellect releases Prime Mega MoE, a trainable fused MoE kernel

Developments

4 developments

  1. Oct 29, 1:41 PM ET · 1 article
    Prime Intellect compares Prime Mega MoE per-layer speed against prime-rl baseline
    Prime Intellect: Prime Intellect's Mega MoE speeds up MoE layers up to 2.47×
  2. Oct 29, 1:41 PM ET · 1 article
    Prime Intellect compares Prime Mega MoE with prime-rl baseline on training speed
    Prime Intellect: Prime Intellect's Mega MoE trains Qwen3 models 32–34% faster
  3. Oct 29, 1:41 PM ET · 2 articles
    Prime Intellect introduces Prime Mega MoE, a fused forward and backward MoE kernel
    MarkTechPost: Prime Intellect releases Prime Mega MoE, a trainable fused MoE kernel
  4. Oct 29, 1:41 PM ET · 2 articles
    Prime Intellect reports prime-rl with Mega MoE trains 48% and 55% faster than TorchTitan
    Prime Intellect: Prime Intellect makes DeepSeek's Mega MoE kernel trainable

Article timeline

The articles in this story. Times are ET.

Oct 29
  1. MarkTechPostNews
    Prime Intellect releases Prime Mega MoE, a trainable fused MoE kernel

    AIPrime Intellect has released Prime Mega MoE, a training extension of DeepSeek's Mega MoE kernel that adds a fused, overlapping backward pass. Self-reported tests on Qwen3 MoE models show full training steps 32% to 34% faster than the prime-rl baseline and 48% to 55% faster than TorchTitan, with the forward pass up to 3.11x faster on a 30B layer. The kernel supports BF16, MXFP8 on Blackwell, and blockwise FP8 on Hopper, and ships in prime-rl as the mega_moe dispatch backend under Apache 2.0.

  2. Prime IntellectOfficial
    Prime Intellect's Mega MoE speeds up MoE layers up to 2.47×

    AIPrime Intellect says its Prime Mega MoE runs the forward pass up to 3.11× faster and the backward pass up to 1.87× faster than its prime-rl baseline per MoE layer. With activation checkpointing, the full layer is up to 2.47× faster.

    Image from @PrimeIntellect's post
  3. Prime IntellectOfficial
    Prime Intellect's Mega MoE trains Qwen3 models 32–34% faster

    AIPrime Intellect says its Prime Mega MoE cuts training step time by 34% on Qwen3-30B-A3B compared with its prime-rl baseline. On Qwen3-235B-A22B, the company reports a 32% faster training step.

    Image from @PrimeIntellect's post
  4. Prime IntellectOfficial
    prime-rl with Mega MoE trains Qwen3 MoE models faster than TorchTitan

    AIPrime Intellect says prime-rl with Mega MoE completes training steps 48% faster than TorchTitan on Qwen3-30B-A3B. On Qwen3-235B-A22B, the same comparison shows steps 55% faster.

    Image from @PrimeIntellect's post
  5. Prime IntellectOfficial
    Prime Intellect releases Prime Mega MoE with overlapping backward kernel

    AIPrime Intellect says Prime Mega MoE fuses MoE routing, dispatch, compute, and return into fast overlapping kernels that keep GPUs from waiting on communication. It extends DeepSeek's Mega MoE, which fuses these steps in the forward pass, by adding a backward kernel that covers both passes.

    Image from @PrimeIntellect's post
  6. Prime IntellectOfficial
    Prime Intellect makes DeepSeek's Mega MoE kernel trainable

    AIPrime Intellect says it made DeepSeek's Mega MoE kernel trainable. On two Qwen3 models, it reports up to 55% faster MoE training than TorchTitan and up to 34% faster hosted training with prime-rl.

    Video from @PrimeIntellect's post

Heat Trend

Not enough continuous observations to show a trend yet.