Skip to content
View original post on X: Ruixiang ZhangX· 42/100AI score42/100

Apple ML Research launches RSI-Inference Bench for AI agents improving inference systems

AISummary

Apple ML Research introduces RSI-Inference Bench, a benchmark built from real vLLM and SGLang tasks to test whether AI agents can improve the systems that run AI models.

The benchmark covers day-0 model support, inference optimization, and bug fixes, with binary and continuous verifiable rewards. The best agent currently passes 26.7% of tasks at pass@1, and the benchmark is described as unsaturated.

Post on XView on X
Ruixiang ZhangVerified on X
@onloglogn

Introducing RSI-Inference Bench from 🍎 Apple ML Research
🌐 http://rsi-inference.com
💻 https://github.com/apple-aiml-research/ml-rsi-inference-bench

Can AI agents improve the systems that run AI?

We spent $100K API tokens, 50K GPU hours to find out. We built environments and benchmark from real-world #vLLM and #SGLang tasks, covering the full lifecycle of model inference engineering:

🚀 Day-0 Model Support
⚡ Inference Optimization
🔧 Bug Fix

→ Unsaturated: best agent at 26.7% pass@1
→ Comprehensive coverage of the inference stack
→ Verifiable rewards: binary and continuous

Watch the video to find out more!

Source: Ruixiang Zhang · x.comPublished · added here

Topics

AI agentsDeployment & engineeringPapers & researchEvals & benchmarks