🚀 Introducing Δτ-Bench: Pushing Agents Toward a Moving Capability Frontier!
📄 Paper: http://arxiv.org/abs/2610.34238
💻 Code: https://github.com/zjunlp/delta-tau-bench
🤗 Dataset: https://huggingface.co/datasets/zjunlp/delta-tau-bench
Tools evolve. Does your agent keep up?
APIs change, familiar tools disappear, and better alternatives emerge. An agent may still complete a task while missing a much more efficient solution.
🌍 Δτ-Bench evaluates whether agents can keep pace with evolving tool ecosystems:
- Executable, multi-turn tasks across diverse domains.
- Successive ecosystem changes: interfaces adapt, functions split, and workflows merge.
- Evaluation of task success, execution efficiency, recovery from disrupted workflows, and discovery of better alternatives.
📊 What do we find?
• Models struggle to reach the current frontier. Agents have difficulty identifying relevant, currently valid tools and using them efficiently. Successful execution can still hide missed opportunities.
• Skill-based methods struggle to track the moving frontier. Accumulated skills help initially, but their benefits do not consistently survive ecosystem changes. Better task performance does not necessarily mean better adaptation, and skill maintenance can be costly.
🛠️ We introduce STEWARD, an ecosystem manager that jointly manages tools and procedural knowledge through:
• Incremental skill maintenance: tracking skill–tool dependencies and selectively updating skills with verified experience while preserving reusable knowledge.
• Dynamic capability provisioning: jointly supplying relevant tools and skills, then expanding them as new requirements or execution feedback reveal capability gaps.
🔍 Three takeaways from our STEWARD experiments:
• Stronger adaptation with lower overhead. STEWARD improves task success, efficient execution, shortcut discovery, and workflow recovery while reducing maintenance costs.
• Preserving and updating experience pays off. Incremental maintenance better balances knowledge reuse and adaptation than freezing the skill library or rebuilding it from scratch.
• Tool and skill search are complementary. Finding available tools and knowing how to use them work best together; their joint provision improves performance beyond either alone.
Can your agent keep learning as its tools change? Try Δτ-Bench!