# Stanford and Google DeepMind propose TURTLE to evolve harder tool-use benchmark tasks

> Original title: New Stanford and Google DeepMind paper shows that you can turn too-easy tool-use benchmarks into hard, still-solvable tests by evolving e...

- Source: Rohan Paul
- Published: 2026-11-08T23:36:48.000Z
- Bad Influence: https://badinfluence.news/items/o7chjg7vl69crad104m0va58r
- Original: https://x.com/rohanpaul_ai/status/2119574891016135029

## Summary

Stanford and Google DeepMind researchers present TURTLE, a method that uses AlphaEvolve to mutate existing tool-use tasks toward about 50% solver success. In their AppWorld tests with Gemma 4 31B-it, 69% of original tasks were always solved, while 52.5% of evolved tasks were rated hard by an LLM judge, versus 19.8% of originals.

## Post

New Stanford and Google DeepMind paper shows that you can turn too-easy tool-use benchmarks into hard, still-solvable tests by evolving existing tasks against the model.

Tool-use benchmarks like AppWorld are static, so models soon solve most tasks, and building new ones takes human work.

TURTLE uses AlphaEvolve to mutate existing tasks, adding filters, extra steps and hidden lookups, then runs the solver model 10 times on each. Tasks score highest when the solver succeeds about 50% of the time, and each must be solved at least once.

69% of original AppWorld tasks were always solved by Gemma 4 31B-it, while nearly all 735 evolved tasks sat at 40% to 60% completion. An LLM judge rated 52.5% of evolved tasks hard, versus 19.8% of the originals.

If your model aces a benchmark, evolve its tasks toward a 50% success rate instead of writing a new benchmark from scratch.

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHWo-C_paIAEdND2.jpg&mode=thumb&exp=1794528000&sig=8c0297393a8b622c)