Apple researchers present tri-modal masked diffusion model pretrained from scratch
Overview
Apple Machine Learning Research reports the first tri-modal Masked Diffusion Model pretrained from scratch on text, image-text, and audio-text data.
The authors analyze multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects. They also propose an SDE-based reparameterization that separates physical batch size from logical batch size.
A preliminary 3B-parameter model trained on 6.4 trillion tokens performs strongly on text generation, text-to-image, and text-to-speech tasks, according to the report. The report does not give benchmark scores.
Written by AI from the articles below · updated Oct 14, 12:31 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- Apple Machine Learning ResearchOfficialApple researchers pretrain a tri-modal masked diffusion model from scratch
AIApple Machine Learning Research reports the first tri-modal Masked Diffusion Model pretrained from scratch on text, image-text, and audio-text data. The authors analyze multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects, and propose an SDE-based reparameterization that decouples physical from logical batch size. A preliminary 3B-parameter model trained on 6.4T tokens performs strongly on text generation, text-to-image, and text-to-speech tasks.
Heat trend
Not enough continuous observations to show a trend yet.