Skip to content
Trending storyDeveloping

Artificial Analysis says Mercury 3 AA-Omniscience and GDPval-AA v2 scores are not comparable

1 article1 sourcesince Oct 19Last article 1h ago ·

Overview

AISummary of 1 article

Artificial Analysis says scores for Inception's Mercury 3 cited in a recent announcement are not part of its Intelligence Index and are not comparable to its results.

The announcement referred to AA-Omniscience and GDPval-AA v2 scores, which Artificial Analysis says do not follow its methodology.

Artificial Analysis says AA-Omniscience uses a private dataset, of which it open sourced a 10% public subset. It says GDPval-AA v2 would need its grading methodology and its model comparison set to be comparable.

Written by AI from the articles below · updated Oct 19, 3:17 PM ET

Check the sources:

Latest developmentOct 19, 3:04 PM ET
Artificial Analysis says Inception's Mercury 3 scores are not comparable to its index

Article timeline

The articles in this story. Times are ET.

Oct 19
  1. Artificial AnalysisOfficial
    Artificial Analysis says Inception's Mercury 3 scores are not comparable to its index

    AIArtificial Analysis says a recent announcement cited AA-Omniscience and GDPval-AA v2 scores for Inception's Mercury 3 that are not part of its Intelligence Index. It says AA-Omniscience uses a private dataset, with only a 10% public subset open sourced, and GDPval-AA v2 must use its grading methodology and model comparison set to be comparable.

Heat Trend

Not enough continuous observations to show a trend yet.