Together AI launches Together Compiler, agent-driven continuous inference optimization
Overview
Together AI introduces Together Compiler (TKCC), a system that uses AI agents to keep tuning inference as traffic and hardware change.
The company says TKCC reaches up to 2.6x higher tokens per second across frontier open-source models, with 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash on SemiAnalysis' InferenceX low-latency benchmark. It also reports 1.7x higher TPS than open-source megakernels on Llama 1B at batch size 1.
The company presents the work as part of Polymorphic Inference, its approach to serving open models whose architectures and traffic shapes change quickly. Together Compiler is in limited beta for end customers, by request. Together AI says customers will later be able to use it to optimize their dedicated endpoints for their own workloads, with no date given.
Written by AI from the articles below · updated Nov 4, 10:43 AM ET
Check the sources:
Developments
2 developments
- Nov 3, 7:00 PM ET · 1 articleTogether AI explains polymorphic inference and continuous compilationTogether AI Blog: Together AI explains polymorphic inference and continuous compilation for open models
- Nov 3, 7:00 PM ET · 2 articlesTogether AI introduces research on continuous compilation for AI inferenceTogether AI Blog: Together AI introduces Together Compiler for continuous AI inference optimization
Article timeline
The articles in this story. Times are ET.
- Together AI BlogOfficialPickTogether AI introduces Together Compiler for continuous AI inference optimization
AITogether AI introduces Together Compiler (TKCC), a continual optimization loop that uses agents to tune inference for changing traffic and hardware. The company says TKCC reaches up to 2.6x higher TPS across frontier open-source models on SemiAnalysis' InferenceX AgentX benchmark, and 1.7x over open-source megakernels on Llama 1B at batch size 1. It also reports 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash at low latency.
- Together AI BlogOfficialPickTogether AI introduces Together Compiler for continuous AI inference optimization
AITogether AI introduces Together Compiler (TKCC), a system that uses agents to continuously optimize AI inference as traffic and hardware change. The post reports up to 2.6x higher TPS across frontier open-source models and 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash on the InferenceX low-latency benchmark. It also reports 1.7x higher TPS than open-source megakernels on Llama 1B at batch size 1.
- Together AI BlogOfficialTogether AI explains polymorphic inference and continuous compilation for open models
AITogether AI describes Polymorphic Inference, an approach to serving open-source models whose architectures and traffic shapes change quickly. The company says it uses AI agents to optimize their own inference through Continuous Compilation, with the Together Compiler in limited beta for early-access customers.
Heat Trend
Not enough continuous observations to show a trend yet.