Skip to content
Trending storyMonitoring

Artificial Analysis ranks Base Labs models highest on AA-Symmetry consistency benchmark

1 article1 sourcesince Nov 8Last article

Overview

AISummary of 1 article

Artificial Analysis introduces AA-Symmetry, a benchmark of 120 base questions and 440 variations that tests whether models give consistent answers to equivalent questions.

Base Labs models score highest, with Mammoth 27B at 96 and Elephant 9B at 93. Artificial Analysis attributes the Base Labs lead partly to high abstention rates.

Mammoth 27B selects the same answer across variations on 99% of questions, compared with 91% for Drex Max v1.1, 88% for GLiDE v1.2 and 67% for Clef, according to the same report.

Written by AI from the articles below · updated Nov 9, 4:23 PM ET

Check the sources:

Latest developmentNov 8, 6:51 PM ET
Artificial Analysis AA-Symmetry ranks Base Labs models highest on consistency

Article timeline

The articles in this story. Times are ET.

Nov 8
  1. Artificial AnalysisOfficial
    Artificial Analysis AA-Symmetry ranks Base Labs models highest on consistency

    AIArtificial Analysis introduces AA-Symmetry, a benchmark of 120 base questions and 440 variations that tests whether models give consistent answers to equivalent questions. Base Labs models lead, with Mammoth 27B scoring 96 and Elephant 9B scoring 93, driven by high abstention rates. Mammoth 27B selects the same answer across variations on 99% of questions, compared with 91% for Drex Max v1.1, 88% for GLiDE v1.2 and 67% for Clef.

    Image from @ArtificialAnlys's post

Heat Trend

Current comparable heat
5
Peak comparable heat
5Nov 9, 5:00 PM ET
Change in past 24 hours
–

The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.