Researchers propose reward design to keep AI agents in check without inspecting them

Stephen Morris, Elad Hazan and colleagues propose mechanism design, which sets rewards so AI agents behave as intended without requiring researchers to understand their internal workings, according to MIT Technology Review.

Morris's preprint suggests rewarding a model for how well its answers predict a second model's output, so that sandbagging on tests becomes a losing bet. Hazan proposes agents that alternate as solvers and auditors, with cheating solvers punished and auditors punished for failing to catch them. Hazan's approach still faces the risk that agents collude.