Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date.
It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with human judgment. We do this by painstakingly assigning severity scores to 330+ error types, and tuning each metric to match the severity scores.
Existing benchmarks either use brittle deterministic metrics (e.g. OmniDocBench) or LLM-as-a-judge (ParseArena), which can help recognize equivalent representations but are coarse/stochastic/subject to bias by the prompt+model.
ParseBench-2 also contains:
✅ 4.3k+ document pages (double from ParseBench-1).
✅ An extra dimension (forms)
✅ Much better ground-truth across all dimensions compared to ParseBench-1.
This is the next evolution of ParseBench, our hugely popular benchmark that has 50k+ monthly downloads, over 150+ OCR solutions benchmarked, and has been adopted by companies like NVIDIA, Cohere, Aleph Alpha, Perplexity, and others.
Alongside ParseBench-2, we are also releasing OCRia-1.5 - a new family of document parsing models that powers all our modes in LlamaParse. They represent a massive step up in capabilities from our latest release, and are state of the art in parsing complex tables, forms, charts, formatting, and assigning bounding boxes to complex layouts.
Come check out our blog post, technical report, and website.
Blog: https://www.llamaindex.ai/blog/introducing-parsebench-v2?utm_source=socials_jerry_liu&utm_medium=li_social&utm_campaign=launch&utm_content=blog_post
Technical Report: https://www.llamaindex.ai/files/parsebench-v2-severity-aware-evaluation-of-document-parsing.pdf?utm_source=socials_jerry_liu&utm_medium=tw_social&utm_campaign=launch&utm_content=technical_report
Website: https://www.parsebench.ai/
If you’re interested in trying out our OCRia-1.5 capabilities, come sign up for LlamaParse: https://cloud.llamaindex.ai/