Skip to content
Trending storyMonitoring

Amazon paper finds errors in AI rule-following benchmarks

1 article1 sourcesince Nov 8Last article

Overview

AISummary of 1 article

An Amazon paper says public benchmarks that test AI models on tax, legal and business rules often mark correct answers wrong, because their answer keys or grading code contain mistakes.

The authors audited nine benchmarks. They report that LegalBench's tax scorer deleted decimal points, reading $259,487.13 as 25,948,713. After fixing the grading, the number of accepted answers for each of three models rose from 8 or 9 to between 87 and 91 out of 96.

Written by AI from the articles below · updated Nov 9, 9:25 PM ET

Check the sources:

Latest developmentNov 8, 4:24 PM ET
Amazon paper finds errors in AI rule-following benchmarks

Article timeline

The articles in this story. Times are ET.

Nov 8
  1. Rohan PaulX
    Amazon paper finds errors in AI rule-following benchmarks

    AIAn Amazon paper finds that public benchmarks testing AI on tax, legal and business rules often mark correct answers wrong because their answer keys or grading code contain mistakes. The authors audited nine benchmarks and found LegalBench's tax scorer deleted decimal points, reading $259,487.13 as 25,948,713. Fixing it raised accepted answers from 8 or 9 to between 87 and 91 out of 96 for each of three models.

    Image from @rohanpaul_ai's post

Heat Trend

Not enough continuous observations to show a trend yet.