Skip to content
Trending storyDeveloping

Mattt publishes Swift eval finding Sosumi lifts coding agent pass rates to about 98%

1 article1 sourcesince Nov 11Last article

Overview

AISummary of 1 article

Mattt publishes an evaluation of Sosumi, his sosumi.ai documentation service, on Swift API coding tasks, reporting that agents using it pass about 98% of tasks.

The eval has 41 Swift tasks, mostly covering new APIs in the OS 27 SDKs, and each attempt is checked by the compiler and per-task rules.

Mattt says sosumi.ai served tens of millions of requests from coding agents in 30 days, from about 6,900 unique IPs per day.

Written by AI from the articles below · updated Nov 11, 12:38 PM ET

Check the sources:

Latest developmentNov 11, 9:58 AM ET
Mattt's Swift API eval finds Sosumi lifts coding agent pass rates to about 98%

Article timeline

The articles in this story. Times are ET.

Nov 11
  1. MatttX
    Mattt's Swift API eval finds Sosumi lifts coding agent pass rates to about 98%

    AIMattt says the sosumi.ai docs service served tens of millions of requests to coding agents in 30 days, from about 6,900 unique IPs per day. He built a 41-task Swift eval, mostly on new OS 27 SDK APIs, checked by the compiler and per-task rules.

    Two bar charts from the Swift API eval: 41 Swift tasks, mean of 3 runs per task. Each model has three conditions: no docs, built-in web search, and Sosumi. Tasks passed: Claude Opus 5.5, 38%, 84%, and 98%. GPT-6.1 Sol, 33%, 99%, and 97%. Claude Sonnet 5.5, 31%, 63%, and 94%. Claude Haiku 5.5, 23%, 64%, and 76%. API cost per task, in US cents: Claude Opus 5.5, 2.5, 11.2, and 5.7. GPT-6.1 Sol, 1.5, 4.0, and 1.3. Claude Sonnet 5.5, 0.8, 4.5, and 2.2. Claude Haiku 5.5, 0.1, 2.0, and 0.2.

Heat Trend

Current comparable heat
5
Peak comparable heat
9Nov 11, 1:00 PM ET
Change in past 24 hours
–

The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.