Google Cloud shares four evaluation lessons from building its Developer plugin

Google Cloud's Developer plugin team says LLM judges that grade an agent's final text miss what it actually did, so they score raw tool-call trajectories instead.

The team also says it runs live and plan-only evaluations side by side, because live runs can be blocked by permission errors before the behavior under test appears, and it recommends golden sets curated by human experts as durable baselines.