# Mattt's Swift API eval finds Sosumi lifts coding agent pass rates to about 98%

> Original title: Over the last 30 days, http://Sosumi.ai has served tens of millions of requests to coding agents looking up Apple docs, from about 6,900 ...

- Source: X search: indie AI launches (I built, solo founder)
- Published: 2026-11-11T14:58:07.000Z
- Bad Influence: https://badinfluence.news/items/iox5f5mc35ov48o5b9qyy3so2
- Original: https://x.com/mattt/status/2120531522562511020

## Summary

Mattt says the sosumi.ai docs service served tens of millions of requests to coding agents in 30 days, from about 6,900 unique IPs per day. He built a 41-task Swift eval, mostly on new OS 27 SDK APIs, checked by the compiler and per-task rules.

## Post

Over the last 30 days, http://Sosumi.ai has served tens of millions of requests to coding agents looking up Apple docs, from about 6,900 (nice) unique IPs/day.

But does it actually help? I built an eval to find out: 41 Swift tasks, mostly new APIs in the OS 27 SDKs, checked by the compiler and per-task rules.  

Without docs, Opus, GPT-6.1 Sol, and Sonnet pass about a third of them. With Sosumi, they pass 98%, 97%, and 94%. 

Compared to built-in web search, Claude models score 12 to 30 points higher with Sosumi, at about half the cost or less. GPT does slightly better with web search (99%), but at 3× the cost.  

Running the eval also turned up a handful of fixes, now live (one framework page was 3 MB 😬). The eval is open source: https://github.com/NSHipster/sosumi.ai/tree/main/evals/swift-api

![Two bar charts from the Swift API eval: 41 Swift tasks, mean of 3 runs per task. Each model has three conditions: no docs, built-in web search, and Sosumi. Tasks passed: Claude Opus 5.5, 38%, 84%, and 98%. GPT-6.1 Sol, 33%, 99%, and 97%. Claude Sonnet 5.5, 31%, 63%, and 94%. Claude Haiku 5.5, 23%, 64%, and 76%. API cost per task, in US cents: Claude Opus 5.5, 2.5, 11.2, and 5.7. GPT-6.1 Sol, 1.5, 4.0, and 1.3. Claude Sonnet 5.5, 0.8, 4.5, and 2.2. Claude Haiku 5.5, 0.1, 2.0, and 0.2.](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHW2j8PibIAEVNSO.jpg&mode=thumb&exp=1794700800&sig=f5823ec0c1105846)