# Jerry Liu distinguishes frontier intelligence from specialized cost-accuracy benchmarks

> Original title: There are two main types of benchmarks:

- Source: Jerry Liu
- Published: 2026-11-04T01:14:06.000Z
- Bad Influence: https://badinfluence.news/items/onhbmb1aezkglx2qw9fjxggc0
- Original: https://x.com/jerryjliu0/status/2117787436675997769

## Summary

Jerry Liu says AI benchmarks fall into two types: frontier intelligence tests like TerminalBench and GDPval, where accuracy matters most, and specialized tests where the cost-accuracy Pareto curve matters more. He argues that as frontier benchmarks saturate, they become specialized ones, and that frontier models at xhigh likely saturate specialized tasks but at prohibitive cost and latency.

## Post

There are two main types of benchmarks: 

1. those that measure frontier intelligence (terminalbench, gdpval, etc.). accuracy matters more than cost.

2. those that measure specialized intelligence. the pareto curve of cost/accuracy more than accuracy alone. 

both are important, but they are distinct benchmark categories. 
* as (1) saturates, they become (2). 
* the largest frontier models at xhigh likely already saturate (2), but they do so at a cost/latency point that's completely price-prohibitive.
* every user should define (2) for the tasks they care about the most.  

we're big fans of both. we have so far dedicated outsized attention to (2) especially related to document tasks.
we fundamentally think users should have the most price-efficient tools to understand, extract, and read over their documents. we've built parsebench, extractbench for this purpose. 

we are starting to invest in (1) as well, because certain document tasks are so challenging that even frontier models struggle (e.g. see extractbench pro). we are continuing our investment in long-horizon workflow research at @llama_index