Skip to content
Trending storyDeveloping

Continual-ARC benchmark finds agent harnesses don't learn; training a 9B model does

1 article1 sourcesince Nov 11Last article

Overview

AISummary of 1 article

Daniel Lenton of Unify says that three agent harnesses, OpenClaw, Hermes Agent and Prime Agent, do not meaningfully learn from experience.

On his new Continual-ARC benchmark, none of the three memorized anything of value or generalized to new tasks. Training the weights of a small 9B model did both, according to the same report.

Elvis Saravia, summarizing the result on X, argues that the harness is increasingly part of training pipelines for continual learning, and that it can stay minimal and efficient.

Written by AI from the articles below · updated Nov 11, 5:05 PM ET

Check the sources:

Latest developmentNov 11, 5:04 PM ET
Continual-ARC benchmark finds agent harnesses fail to learn, while training a 9B model succeeds

Article timeline

The articles in this story. Times are ET.

Nov 11
  1. elvisX
    Continual-ARC benchmark finds agent harnesses fail to learn, while training a 9B model succeeds

    AIDaniel Lenton of Unify says OpenClaw, Hermes Agent and Prime Agent do not meaningfully learn from experience. On the new Continual-ARC benchmark, none of the three memorized anything of value or generalized to new tasks, while training a small 9B model's weights did both. Elvis Saravia argues the harness is increasingly part of training pipelines for continual learning, though it can stay minimal and efficient.

Heat Trend

Current comparable heat
7
Peak comparable heat
10Nov 11, 6:00 PM ET
Change in past 24 hours
–

The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.