Harrison Chase outlines a production loop for judging and fixing model failures

Harrison Chase says a judge run on production traffic matters most for what happens to the outputs it flags.

In his description, the loop calibrates the judge against human labels, flags low-scoring outputs, confirms which are real failures, adds those to evals, fixes them, and reruns the evals. He argues that production surfaces failures that an offline dataset never contains. The source is a post on X by Chase, the LangChain co-founder; the report gives no measured results from applying the loop.