LangSmith's Jev judge scores agent traces on difficulty and correctness in one pass
Overview
Harrison Chase, of LangChain, says that trajectory labeling should be split into several questions rather than a single pass/fail judgment.
He describes how Jev, LangChain's judge in LangSmith Evals, works: difficulty and correctness each receive their own typed answer, produced in one pass, on every trace. The report comes from Chase's own post on X, so it describes the design as he presents it rather than independent testing of Jev's accuracy.
Written by AI from the articles below · updated Oct 9, 3:43 AM ET
Check the sources:
Article timeline
Follow the coverage from different perspectives. Times are ET.
- Harrison ChaseLangSmith's Jev judge scores agent trajectories on difficulty and correctness
AIHarrison Chase says trajectory labeling should be split into several questions rather than one pass/fail judgment. LangChain's Jev judge in LangSmith Evals answers difficulty and correctness as separate typed outputs in a single pass on every trace.
Heat trend
Not enough continuous observations to show a trend yet.