Apple shows how to run an 816B-parameter model across Mac Studios with MLX and RDMA
Original titleLocal and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
AISummary
Apple's machine learning research team says MLX can run Muse Spark Open, an 816B-parameter mixture-of-experts model, across four Mac Studios with M5 Ultra at over 36 tokens per second.
The post compares pipeline and tensor parallelization over Thunderbolt RDMA and gives the mlx.launch commands. It also reports TTFT and decode speedups over the M3 Ultra for Qwen 3.8 and Mistral Medium 3.5.
AIWhy it matters
The post gives concrete benchmarks and commands for running a roughly 816B-parameter model across Mac Studios, useful for anyone weighing local inference setups.
Source: Apple Machine Learning Research · machinelearning.apple.comPublished · added here