Skip to content
View original post on X: SantiagoX· 38/100AI score38/100

Constrained Decoding Models skip token generation for classification and extraction

AISummary

Santiago (@svpino) says the white paper's classification and span extraction use a task-specific head that reads the input's internal representation and selects a label or source span.

Classification returns labels from a supplied set, and extraction returns input text with source offsets and confidence scores. The model can still pick a wrong label or passage, but restricting the answer space makes its outputs easier to validate.

Post on XView on X
SantiagoVerified on X
@svpino

There are 4 specialized execution paths here:

  • Classification
  • Extraction
  • Summarization
  • Context Compression

The architecture behind classification and span extraction in their white paper is really cool:

They have a task-specific head that reads the input's internal representation and selects a label or source span. This avoids autoregressive token generation for the prediction itself.

As a result, classification returns labels from the supplied set. Extraction returns text from the input, along with source offsets and confidence scores.

The model can still choose the wrong label or passage, but restricting the answer space makes its outputs easier to validate.

Neal Patel@neal_k_patel
You already gave the LLM the answer. So why is it writing it out from scratch? Introducing a new AI model architecture: Constrained Decoding Models (CDMs) LLMs were built to write. Most production workloads just need an answer: fast, consistent, and affordable. CDMs are purpose-built for 1 task each. They read your input once and skip the token-by-token generative loop that makes LLMs slow and expensive. Not just a decision model, but a family of 3 CDMs live today: • Classify: define your labels and get back a calibrated probability + reason for each. Used for routing, tagging, and intent detection. • Extract: describe the entities in natural language and get back well-formed JSON with confidence scores + citations. Used for web scraping, document processing, and PII redaction. • Summarize: pass in long text, images, or PDFs and get back a consistent summary under 1,000 tokens. Used for context compaction, metadata generation, and transcript digests. Built for production: • 1M tokens of multimodal (text + image) input, 31x larger than Jev • 77.2% on DecisionBench vs Jev's 72% • 200x faster, 467x cheaper than frontier LLM APIs • $0.05/M input tokens. Output is free (yes, even for summarize) Live on your favorite platforms today: @OpenRouter @vercel_dev @LiteLLM @LangChain @TrustedRouter @itsArthurAI @useRouterPlus Get 50M input tokens for free (for the first 50 hours) at link below 👀
View quoted post on X

Source: Santiago · x.comPublished · added here

Topics

Model releasesProduct updatesTrends