How to Do LLM Evals with Decision Models: A Complete Practice from Threshold Tuning to Probability Calibration
An observation from @HamelHusain & @doesdatmaksense: when a batch of new "decision models" such as Jev, Decider, and the Decisions API are used as cheap LLM Judges for automated evaluation, most practitioners simply stick with the default 0.50 threshold and never verify it. This article uses real production data to show that if you bring over the classic classifier evaluation methodology (threshold tuning, precision/recall tradeoffs, probability calibration, cost-performance Pareto analysis), you can squeeze out significant and quantifiable performance gains at no extra cost.
https://hamel.dev/blog/posts/decision-model-evals/
Background: A Problem Is Quietly Happening
A wave of decision model products has recently emerged, including TypeSafe's Jev, Perplexity's Decider, Liquid's d1, Cloudflare's Clef, and OpenAI's Decisions API. They promise "fast, cheap classification" and are seen as low-cost alternatives to LLM Judges. Unlike LLM Judges, which output a binary pass/fail verdict, decision models output a 0–1 failure probability score for each trace.
The problem is that these models' output score distributions differ greatly, yet many people treat 0.50 as a natural dividing line. The author uses an extreme example to illustrate why this is dangerous: suppose a model gives all failing samples 0.40 and all passing samples 0.10. That model is in fact a perfect classifier, but at a 0.50 threshold it catches zero failures. In reality the score distributions of the two classes overlap, and choosing a threshold is essentially a tradeoff between precision and recall, with the right balance point depending on the business scenario (alerting systems require both to be ≥80%; cancer screening favors recall; criminal convictions favor precision).
Experimental Design
· Data: 98 real production traces from an apartment-rental AI agent (61 known failures + 37 non-failures), with the task of scoring for the broad question "is there any substantive problem?"
· Comparison targets: five decision models, using equivalent evaluation-criteria prompts.
· Methodological highlights: stratified five-fold cross-validation, with all models using the same data split; thresholds are selected on the training folds and measured on the held-out folds; all tuning and calibration reuse saved scores, with zero additional model calls, meaning the marginal cost of the entire analysis framework is nearly zero.
· The broad task was kept deliberately: "any problem counts" was chosen to amplify differences between models; in actual production, they recommend one separate judge per failure mode.
Four Key Findings
1. Threshold tuning: a free and necessary gain. Models that already met the bar (Perplexity, Liquid d1-ultra, OpenAI) benefited little from tuning, but Liquid d1 and Clef Flash, which initially fell short, gained noticeably. This shows that eliminating a model without tuning its threshold may wrongly discard it.
2. Cost-performance Pareto tradeoff. Perplexity is cheapest (0.82 cents per 100 evaluations, F1 = 0.79), and OpenAI Decisions is best (F1 = 0.87) but about 4.5 times more expensive (3.72 cents). The author admits the three-point Pareto chart is only a demonstration of the framework, but the conclusion is general: harder evaluation tasks are worth paying for LLMs, which echoes Randy Olson's comparative study (decision models match LLMs at low cost on classification/routing tasks, but LLMs clearly lead on knowledge and reasoning tasks).
3. Probability scores are not necessarily trustworthy and need calibration. A score of 0.80 should mean that about 80% of similar traces actually fail, but reliability charts show that some models are seriously miscalibrated. Clef Flash has a group of samples where the predicted failure probability is only about 8%, yet the actual failure rate is as high as 60%. Platt scaling (fitting a sigmoid to the log-odds) fixed three models, but OpenAI got worse (Brier 0.159 → 0.164); fitting directly to the probabilities instead improved it to 0.154. This is itself an important lesson: the choice of calibration method also affects results and must be measured empirically.
4. Prompts still have room for improvement. A small manual experiment (replacing Jev's simple prompt with a list of failure types) raised F1 from 0.78 to 0.80, showing that before turning to sophisticated methodology, clearly specifying the evaluation criteria remains the highest cost-effectiveness investment.
The Author's Practical Recommendations (Condensed)
1. First set the minimum precision/recall bar for your scenario, and eliminate models that fail to reach it;
2. Tune thresholds before comparing models, and choose the tradeoff that fits business goals; don't just adopt F1 because the article uses it;
3. If you plan to use the scores as confidence, verify calibration and measure whether calibration actually improves things;
4. Weigh performance, cost, and latency together;
5. The whole process is iterative: when the prompt changes, the thresholds and calibration may need to be redone.