Skip to content
Trending storyMonitoring

Stanford and Google DeepMind present TURTLE, which evolves harder tool-use benchmark tasks

1 article1 sourcesince Nov 8Last article

Overview

AISummary of 1 article

Stanford and Google DeepMind researchers present TURTLE, a method that uses AlphaEvolve to mutate existing tool-use tasks toward about 50% solver success.

In AppWorld tests with Gemma 4 31B-it, 69% of original tasks were always solved, according to the researchers as reported by Rohan Paul on X. An LLM judge rated 52.5% of the evolved tasks as hard, compared with 19.8% of the originals.

Written by AI from the articles below · updated Nov 9, 9:25 PM ET

Check the sources:

Latest developmentNov 8, 6:36 PM ET
Stanford and Google DeepMind propose TURTLE to evolve harder tool-use benchmark tasks

Article timeline

The articles in this story. Times are ET.

Nov 8
  1. Rohan PaulX
    Stanford and Google DeepMind propose TURTLE to evolve harder tool-use benchmark tasks

    AIStanford and Google DeepMind researchers present TURTLE, a method that uses AlphaEvolve to mutate existing tool-use tasks toward about 50% solver success. In their AppWorld tests with Gemma 4 31B-it, 69% of original tasks were always solved, while 52.5% of evolved tasks were rated hard by an LLM judge, versus 19.8% of originals.

    Image from @rohanpaul_ai's post

Heat Trend

Not enough continuous observations to show a trend yet.