https://x.com/i/article/2111172254700572672
Why you need a probability, not a dissertation
Generative and discriminative models solve different problems. Here's why that matters.
By Logan Hennessy, Software Engineer
If you look close enough, you’ll probably see engineers still using LLMs as drop-in classifiers. In this case, they’ll ask a model whether a support ticket is urgent, and get something.
But that's a response, not a decision. And there is more lurking in that difference than meets the eye.
What’s missing when you use classifiers?
A classifier might have returned something like {urgent: 0.83, normal: 0.17}. This object does three jobs a paragraph cannot. It enables:
- A threshold. For example, urgency above 0.9 goes to the on-call pager, below 0.2 to the backlog, and everything between to a human. With "Yes," we lose our dial.
- Abstention. An autoregressive decoder has no way to say “Not sure.” But this is a legitimate classifier response.
- Measurement and verification. If the classifier says 0.8 across a thousand tickets, roughly 800 should be urgent. That means an audit trail you can verify and use to calibrate as needed.
Collapse those probabilities to a text and you lose the ability to tune, defer, or audit.
The hidden tax
Text generation makes sense when the audience is a human. We want every em dash and fake parallel. We gladly pay the penalty of computational costs so that we can parse the results as we would a Wikipedia article.
But in modern backend pipelines, code is the consumer. Humans, we hope, are not manually processing each LLM output by hand. And code is not looking for a conversational partner to help it decide what to stream on a random Tuesday night. It wants typed state so it can branch.
When you ask an autoregressive model to emit "Yes" or "No", you are paying an autoregressive tax:
- Inference overhead: Autoregressive generation is sequential. Every output token depends on the previous token. And the decision space is the entire language. Discriminative models predict a tight set of outputs in a single forward pass.
- Unnecessary serialization: You are forcing a model to serialize a mathematical state into English characters, only so your downstream Python code can parse, strip, or regex it back into a boolean.
SafetyRating - A Gemini example
If you call Gemini, you already get both kinds of output (generated and classified) in the same response.
candidates[].content is generated text. Alongside it is candidates[].safety_ratings[] containing a list of SafetyRating objects:
Abbreviated version of google.genai.types.SafetyRating
SafetyRating is not generated text from an LLM. It is a compilation of the output of several discriminative classification models. These outputs are processed and returned in a typed response. This way, the decision to “block” a piece of content, along with the inherent risk, can be dialed up or down to suit the specific use-case.
Generation and judgment: two mechanisms, one API call. And that is a pattern we should expect to see more of. Generative models propose. Discriminative models decide.
"But what about the logprobs?"
Gemini exposes probabilities for each token it generates in logprobs_result (google.genai.types.LogprobsResult, returned in google.genai.types.Candidate). So reasonable people may justifiably wonder, “Why not take the probability of the ‘Yes’ token and move on?”
Because that is P(token), not P(class). P(token) is the model's confidence in emitting a word. That can shift when labels change, options are reordered, or you just reword the question. For example, change urgent to critical and your probabilities change with it when nothing else has.
So logprobs can be a useful rough signal. But they aren’t something you should count on in your monitoring workflow.
Check whether the decision already exists
Before you train anything, check whether someone already shipped your classifier:
- Cloud Vision AI: Label and object detection with per-label confidence
- Natural Language AI: Content classification and moderation with category confidences
- Document AI: Per-field extraction confidence, with human review built on thresholding it
Each is a fixed label space and a score. If your decision fits cleanly into one of those categories, wire it up!
Domain-specific classification
General-purpose discriminative models give us custom classification without any domain-specific training. We can hand the model an enumeration of values that match our domain — ticket classification, threat detection, sentiment analysis — and define our own classes on the fly.
Zero-shot frameworks depend on broad general knowledge and do well on broad general predictions. But because they have no memory of your domain, you have to pass your rubric and rules in the request every time. When you need to classify specialized jargon or internal enterprise pipelines at scale, training or fine-tuning a model on your own labeled dataset bakes those rules straight into the weights — cutting out the prompt overhead and yielding measurably better results.
The rule of thumb
My software engineering professor’s catchphrase was “it depends.” Because most engineering decisions depend on a variety of factors. Decisions necessitate trade-offs, which is exactly where things get interesting.
On the one hand, you have an approach that is fast to ship, higher cost, and lower accuracy — that’s LLM-as-a-judge. On the other, you have an approach that is slower to ship, low-cost (at runtime), and high-accuracy — that’s training a custom model.
Discriminative models are starting to give us something close to the best of both worlds, though there is always more tuning and cost optimization to unlock when you train a model for your specific use-case.
We are still finding countless use-cases for generative models. Nothing about discriminative models negates the usefulness of generation. But when you need a decision — a label, a score, something to gate on — then you probably want a distribution over a fixed set of outcomes rather than the word “Yes.”