Great paper from Meta with practical tips for tuning LLM judges.
This one is on keeping LLM code judges calibrated and tested on a deployed coding agent.
LLM judges often decide whether an agent's code change is good. But their prompts go stale every time the agent changes.
AutoRefine fixes the prompt using signals teams already have, like whether tests passed and whether the change got merged.
On Meta's deployed agent, agreement between the judge and real outcomes rose from 0.16 to 0.51.
https://arxiv.org/abs/2611.09291
Chat with Paper: https://academy.dair.ai/papers/autorefine-auto-calibrating-code-judges-from-workflow-outcomes-2611.09291