Skip to content
View original post on X: Thomas Wolf
Published
Pick62/100

Thomas Wolf says METR's Anthropic AI R&D report is too inward looking

AIThomas Wolf criticizes a METR report on AI R&D acceleration at Anthropic, saying it counts the lab's compute, hires and internal data and credits about 80% of the speed-up to AI. He argues that much AI-accelerated R&D happens outside the lab, through data and RL environment companies that anticipate model failures and sell to labs.

AIWhy it matters.

The post challenges a METR report's method, arguing that its inward-looking data overstates AI's role in accelerating Anthropic's R&D.

Thomas WolfVerified on X
@Thom_Wolf
View on X

this METR report on AI R&D acceleration at Anthropic is very inward looking imo. it counts the lab's compute, hires and internal data, then credits ~80% of the speed-up to AI

what I see from the outside: labs are becoming schelling points for money, redistributing it to an ecosystem they created to accelerate their own development

it's a symbiosis. the best data and RL environment companies now know where models will fail before the labs do, and the labs just buy what they propose. that feeds back into every lab at once

a lot of the "AI accelerating AI R&D" happens outside the lab

METR@METR_Evals
Since August, a team within METR has worked with internal Anthropic information to assess whether AI itself is accelerating AI development. We are now publishing a report summarizing the results of this risk assessment.
View quoted post on X
Source details

Thomas Wolf · x.comPublished

Original title: this METR report on AI R&D acceleration at Anthropic is very inward looking imo. it counts the lab's compute, hires and internal data, th...

Topics

Data & trainingExpert opinion