Meta presents AutoRefine to recalibrate LLM code judge prompts from workflow outcomes

Meta presents AutoRefine, a method that automatically refines the prompts of LLM code judges using outcome signals such as whether tests passed and whether a change was merged.

Per the report by Elvis Saravia (@omarsar0), agreement between the judge and real outcomes on Meta's deployed coding agent rose from 0.16 to 0.51.

The method targets judge prompts that go stale whenever the coding agent changes, so the judge has to be re-tuned to stay aligned. Teams already collect these outcome signals, which AutoRefine uses to fix the prompt without manual relabeling.