# Artificial Analysis AA-Symmetry ranks Base Labs models highest on consistency

> Original title: AA-Symmetry measures how consistent models are when asked equivalent questions. We take a fixed task set and make changes like reordering...

- Source: Artificial Analysis
- Published: 2026-11-08T23:51:04.000Z
- Bad Influence: https://badinfluence.news/items/ua0c9tatb36rbsmut8hw5xypa
- Original: https://x.com/ArtificialAnlys/status/2119578480274980949

## Summary

Artificial Analysis introduces AA-Symmetry, a benchmark of 120 base questions and 440 variations that tests whether models give consistent answers to equivalent questions. Base Labs models lead, with Mammoth 27B scoring 96 and Elephant 9B scoring 93, driven by high abstention rates. Mammoth 27B selects the same answer across variations on 99% of questions, compared with 91% for Drex Max v1.1, 88% for GLiDE v1.2 and 67% for Clef.

## Post

AA-Symmetry measures how consistent models are when asked equivalent questions. We take a fixed task set and make changes like reordering options, swapping labels, or negating yes/no questions. AA-Symmetry includes 120 base questions and total 440 variations, testing consistency in both the chosen answer and its probability.

Models from @baselabs lead on AA-Symmetry, driven by a high abstention rate. They consistently admit they do not know an answer with high confidence across question variations, earning high consistency scores. Mammoth 27B scores 96 and Elephant 9B scores 93 on AA-Symmetry. Mammoth 27B selects the same answer across variations on 99% of questions, compared with 91% for Drex Max v1.1, 88% for GLiDE v1.2 and 67% for Clef.

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHWpAmeDbUAATEfx.jpg&mode=thumb&exp=1794528000&sig=dc6324edc75f1422)