the useful part of a judge on production traffic is what happens to the outputs it flags
calibrate it on human labels, flag low scorers, confirm the real failures, add them to evals, fix, rerun
prod finds the failures your offline dataset never had
Harrison Chase outlines a production loop for judging and fixing model failures
AIHarrison Chase says a judge on production traffic is most useful for what happens to the outputs it flags. His loop calibrates the judge on human labels, flags low scorers, confirms real failures, adds them to evals, fixes them, and reruns.
Harrison ChaseVerified on X
@hwchase17
Now: @Shopify’s continual learning loop with PyTorch and @vLLM_project takes the stage at #PyTorchCon 2026 with Cody Mazza-Anthony @cmazzaanthony. Using Buyer Profile as an example, Cody walks through how a calibrated judge flags low-scoring outputs in production. Critics diagnose each failure, and a replay with a critic hint is scored again by the judge. Accepted repairs expand and improve the training data. Cody shows how a small specialist model can outperform a frontier model on this task, then turns to optimizing inference through prompt compression. Watch the keynote livestream: https://events.linuxfoundation.org/pytorch-conference-north-america/program/keynote-livestream/
Source details
Harrison Chase · x.comPublished · added here
Original title: the useful part of a judge on production traffic is what happens to the outputs it flags