Google Cloud outlines three audits for detecting bias and errors in LLM judges

Google Cloud says LLM judges, the models used to grade other models' outputs, carry biases including verbosity, position, halo, agreeableness and self-enhancement bias, and can miss up to 75% of errors.

The company recommends three audits to test them. Multi-pass consistency checks measure whether a judge gives stable results across repeated runs. Chance-corrected metrics test how closely a judge agrees with human reviewers, adjusted for agreement that would occur by chance. Multi-judge ensembles with decomposed prompting use several judges and split each evaluation into smaller questions.