Google Cloud outlines three methods to audit LLM judges for bias and errors

Google Cloud says LLM judges carry biases including verbosity, position, halo, agreeableness and self-enhancement bias, and that they can miss up to 75% of errors. It recommends three audits: multi-pass consistency checks to measure internal stability, chance-corrected metrics to validate alignment with human reviewers, and multi-judge ensembles with decomposed prompting.