# Meta presents AutoRefine to calibrate LLM code judges from workflow outcomes

> Original title: Great paper from Meta with practical tips for tuning LLM judges.

- Source: Elvis Saravia
- Published: 2026-11-10T20:35:02.000Z
- Bad Influence: https://badinfluence.news/items/q9o3i1j1l1ylmifjr2367wa5q
- Original: https://x.com/omarsar0/status/2120253922933956644

## Summary

Meta presents AutoRefine, a method that automatically refines LLM code judge prompts using signals such as whether tests passed and whether a change was merged. On Meta's deployed coding agent, agreement between the judge and real outcomes rose from 0.16 to 0.51. The method addresses judge prompts going stale whenever the agent changes.

## Post

Great paper from Meta with practical tips for tuning LLM judges.

This one is on keeping LLM code judges calibrated and tested on a deployed coding agent.

LLM judges often decide whether an agent's code change is good. But their prompts go stale every time the agent changes.

AutoRefine fixes the prompt using signals teams already have, like whether tests passed and whether the change got merged.

On Meta's deployed agent, agreement between the judge and real outcomes rose from 0.16 to 0.51.

https://arxiv.org/abs/2611.09291

Chat with Paper: https://academy.dair.ai/papers/autorefine-auto-calibrating-code-judges-from-workflow-outcomes-2611.09291

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHWyntIfbQAAd0Cf.png&mode=thumb&exp=1794528000&sig=be7945797ea39ca2)