AA-Symmetry measures how consistent models are when asked equivalent questions. We take a fixed task set and make changes like reordering options, swapping labels, or negating yes/no questions. AA-Symmetry includes 120 base questions and total 440 variations, testing consistency in both the chosen answer and its probability.
Models from @baselabs lead on AA-Symmetry, driven by a high abstention rate. They consistently admit they do not know an answer with high confidence across question variations, earning high consistency scores. Mammoth 27B scores 96 and Elephant 9B scores 93 on AA-Symmetry. Mammoth 27B selects the same answer across variations on 99% of questions, compared with 91% for Drex Max v1.1, 88% for GLiDE v1.2 and 67% for Clef.