# Artificial Analysis makes Time per Task its headline API benchmark metric

> Original title: Announcing the next version of Artificial Analysis API Performance Benchmarking: Time per Task is now our headline performance metric, me...

- Source: Artificial Analysis
- Published: 2026-11-04T17:34:29.000Z
- Bad Influence: https://badinfluence.news/items/nofdkzyl4hvf1p83lcpz6hry1
- Original: https://x.com/ArtificialAnlys/status/2118034157909712992

## Summary

Artificial Analysis says it now uses Time per Task as the headline metric for comparing model speed on its roughly 1,000 serverless endpoints. The metric divides output tokens per Intelligence Index task by end-to-end output speed, which counts the full request from send to final token. The source says API default effort cuts Time per Task by about 3 to 5x for GPT-6 Astra, GPT-6.1 Sol and Claude Opus 5.5, with lower Intelligence Index scores.

## Why it matters

The source shows that API default effort settings can cut task time several-fold at a cost to benchmark scores, so default-setting comparisons may differ from headline rankings.

## Post

Announcing the next version of Artificial Analysis API Performance Benchmarking: Time per Task is now our headline performance metric, measuring how long a model takes to complete a task rather than how fast it streams

The update applies to the ~1,000 serverless endpoints we benchmark multiple times a day for speed, latency and caching.

Key changes:
➤ Time per Task is now the headline metric to compare models: as workloads become more agentic, the time to complete a task matters more than speed per token. A model that is fast per token but verbose can take longer than a slower model that uses fewer tokens. Time per Task divides a model's output tokens per Intelligence Index task by its End-to-End Output Speed.

➤ End-to-End Output Speed covers the full request: speed is now measured from sending the request to receiving the final token. That includes the wait before the first token (prefill, and reasoning for models that hide it) as well as generation. Output tokens now come from each provider's API-reported counts instead of the o200k_base tokenizer, and we monitor for discrepancies in reported counts.

➤ Agentic workloads are now part of every measurement: we benchmark short, document-length and long-context prompts, plus multi-turn sessions across coding and knowledge work. Headline figures are the average across these workloads over the past 5 days.

➤ API default effort is now tracked alongside highest effort: most developers run models at the effort level each lab sets as its API default, so we now benchmark key models at that setting as well as at their highest effort. Effort names are not comparable across labs, so we use each lab's own default rather than one shared setting.

Key takeaways:
➤ OpenAI models make up most of the Intelligence vs. Time per Task Pareto frontier despite slower output speeds: across effort levels, GPT-6 Astra and GPT-6.1 Sol hold the frontier from ~45 to ~55 on the Intelligence Index. At max effort, GPT-6 Astra runs at 51 tokens/s but finishes a task in ~17 minutes, against ~26 for Claude Opus 5.5 (max, 111 tokens/s) and ~29 minutes for Gemini 4 Carbon (high, 71 tokens/s), because it uses ~50k output tokens per task against ~120k to ~170k.

➤ API default effort cuts Time per Task by ~3 to 5x and Cost per Task by ~2 to 4x for GPT-6 Astra, GPT-6.1 Sol and Claude Opus 5.5: all three default to medium effort, which scores ~5 to 11 points lower on the Intelligence Index in exchange. Claude Opus 5.5 falls from ~26 minutes and $10.04 per task at max to ~6.8 minutes and $2.45 at medium effort.

➤ Endpoints that optimize for decode speed alone see the largest drop in End-to-End Output Speed: high decode speeds can come at the cost of latency, for example by queuing requests to run at lower batch sizes, or by holding output and sending it in a single burst. Both lengthen the wait before the first token, which End-to-End Output Speed now counts.

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHWTEIudbgAAja9m.jpg&mode=thumb&exp=1794009600&sig=2ce00e4e21a4984d)