Skip to content
Trending storyDeveloping

Garry Tan open-sources g-harness-evals, a benchmark for agent harness design

1 article1 sourcesince Nov 9Last article 17h ago ·

Overview

AISummary of 1 article

Garry Tan has open-sourced g-harness-evals, a benchmark that runs the same frontier model on the same tasks while varying only the agent harness, meaning the system prompts, tools and memory around the model.

According to a post by meng shao on X, pass rates on the 20-task public dev set range from about 71% to 79%, and 14 tasks score identically across all harness combinations.

Written by AI from the articles below · updated Nov 9, 9:27 PM ET

Check the sources:

Latest developmentNov 9, 3:54 AM ET
Garry Tan open-sources g-harness-evals, a benchmark comparing agent harnesses

Article timeline

The articles in this story. Times are ET.

Nov 9
  1. meng shaoX
    Garry Tan open-sources g-harness-evals, a benchmark comparing agent harnesses

    AIGarry Tan has open-sourced g-harness-evals, which holds the frontier model fixed and varies only the agent harness, covering system prompts, tools and memory. On its 20-task public dev set, pass rates range from about 71% to 79%, and 14 tasks score identically across all combinations.

    Image from @shao__meng's post

Heat Trend

Not enough continuous observations to show a trend yet.