# DeepMind paper finds RL-trained moral reasoning gains order consistency, not robustness

> Original title: New paper from Google DeepMind on training models to argue well about moral dilemmas.

- Source: Elvis Saravia
- Published: 2026-10-25T19:59:01.000Z
- Bad Influence: https://badinfluence.news/items/htwgx17fxs4cw5qclrm8wesq9
- Original: https://x.com/omarsar0/status/2114446656032874516

## Summary

Google DeepMind trains a model with reinforcement learning on moral dilemmas from classic novels, using self-judged argument tournaments. The trained model becomes more consistent when answer options are reordered, but user pushback and reframing still change its judgments. The post's author argues that order invariance is easy to optimize and measure without improving underlying judgment, so self-rewarded or consistency metrics should be checked separately for robustness to pressure.

## Post

New paper from Google DeepMind on training models to argue well about moral dilemmas.

The trained model became more decisive and stopped changing its answer when options were reordered. Sycophancy and reframing still moved its judgments.

The setup uses real dilemmas from classic novels, each checked by a logic solver to confirm a genuine conflict. A policy is trained with RL in a self-judged tournament with masked thinking, where it rates pairs of its own arguments on quality. Its ratings agree with human annotators.

The policy then adopts the register it rewards. On MoralChoice, ELEPHANT and MORABLES, it is more consistent across option orders, yet a user pushing back or a reframed question still changes the answer.

Order invariance is easy to optimize and easy to measure, and here it improved without the underlying judgment improving.

If you use self-rewarding or consistency metrics as training signals, check robustness to pressure separately.

https://arxiv.org/abs/2610.30349

Chat with Paper: https://academy.dair.ai/papers/consistency-is-not-validity-self-rewarded-moral-argumentation-makes-language-mod-2610.30349

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHVgGBwebYAAn-BJ.jpg&mode=thumb&exp=1793491200&sig=b3718c6e387be491)