We got OCRia-1 generating tokens nearly 10× faster on a single B200.
Getting there meant making much better use of the same GPU. We tuned how OCRia-1 runs for the demands of document parsing, testing every change for both speed and accuracy.
Some improvements were only 2–5%. Seventeen smaller optimizations later, “just a few percent” starts adding up.
Our infrastructure team wrote up what worked, how they tested the changes, and what they learned about making document models run faster.
We also plan to contribute some of these optimizations back to the open-source vLLM project. Read the full breakdown →
https://www.llamaindex.ai/blog/the-inference-stack-behind-ocria
Watch the video below to see OCRia-1’s token generation before and after the optimizations.