Hamel Husain explains tuning decision thresholds for decision-model evals

Hamel Husain says the default 0.50 decision threshold can leave performance on the table when decision models are used as evals, and that thresholds should be tuned for precision, recall, and F1 before comparing cost.

In a post on using decision models for evals, he tests Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions (GPT-6 Luna) on 100 apartment-leasing agent traces.

The post covers tuning thresholds and calibrating scores for these models, with cost comparisons following the threshold step. The report does not give the resulting scores or threshold values, so the size of any gain from tuning is not stated.