Skip to content
View original post on X: Prime IntellectOfficial
Published
18/100

Prime Intellect says web access lets models find benchmark answers online

AIPrime Intellect says web access is a difficult problem for public evaluations because models can find the answers online. It says HLE-Diamond with tool calls uses a bespoke script to detect circumvention, but models still find ways around it, such as by finding traces online.
Prime IntellectVerified on XOfficial
@PrimeIntellect
View on X

Part of a thread · earlier post

Web access is a gnarly problem for public evaluations, as the answers are accessible to the models.

This is also a problem for HLE-Diamond (with tool calls), which has a bespoke script to detect circumvention.

However, models find ways around it, e.g., by finding traces online:

Source details

Prime Intellect · x.comPublished

Original title: Web access is a gnarly problem for public evaluations, as the answers are accessible to the models.

Topics

Safety & alignmentEvals & benchmarksExpert opinion