Skip to content
Trending storyDeveloping

JetBrains Research proposes multi-turn coding-agent benchmark SWE-TaskFlow

1 article1 sourcesince Oct 12Last article 1h ago ·

Overview

AISummary of 1 article

JetBrains Research argues that coding-agent benchmarks should mirror how developers use agents across a session, including switching between questions, planning, reviews and edits.

The authors studied 4,782 real agent sessions in JetBrains IDEs and built SWE-TaskFlow, which splits SWE-Bench Pro issues into several turns.

In a pilot on 700 tasks, the split roughly doubled agent cost without a stable change in resolve rate, according to the report shared on X by Rohan Paul.

Written by AI from the articles below · updated Oct 12, 4:22 AM ET

Check the sources:

Latest developmentOct 12, 4:17 AM ET
JetBrains Research says coding-agent benchmarks should match real user task flows

Article timeline

The articles in this story. Times are ET.

Oct 12
  1. Rohan PaulX
    JetBrains Research says coding-agent benchmarks should match real user task flows

    AIJetBrains Research argues that coding-agent benchmarks should reflect how developers actually use agents, switching between questions, planning, reviews and edits across a session. The authors studied 4,782 real agent sessions in JetBrains IDEs and built SWE-TaskFlow, which splits SWE-Bench Pro issues into several turns. In a pilot on 700 tasks, splitting roughly doubled agent cost without a stable change in resolve rate.

    Image from @rohanpaul_ai's post

Heat trend

Not enough continuous observations to show a trend yet.