Stanford CS 329Z lectures on agent evaluation: Terminal-Bench, Harbor, benchmark design

Meng Shao's X posts report two Stanford CS 329Z lectures on agent evaluation.

Lecture 10, "Evaluating Agents: Terminal-Bench and Harbor," presents Terminal-Bench as a community benchmark of terminal tasks and Harbor as infrastructure for building custom evaluations. It argues that evaluators are becoming a core engineering role and that evaluations serve as the specification for agent development.

Lecture 13, "Evaluation Fundamentals & Benchmark Design," says an agent benchmark has a task definition and evaluation rules. It lists seven criteria for good benchmarks: challenge, realism, discriminability, ease of running, verifiability, interpretability, and durability. It also names fairness, reproducibility, transparency, and honesty as conditions for trustworthy scores.

These posts report the lectures' framing only. They do not include the lectures' own evidence or results.