OpenAI, Google, Microsoft and Anthropic back a shared robots.txt opt-out for AI scraping
AIOpenAI, Google, Microsoft and Anthropic say they will honor a new shared "policy token" signal that publishers can place in robots.txt to opt out of AI scraping. Publishers could then set AI training preferences once instead of separately for each company.
AIWhy it matters.
The article explains how a shared robots.txt token would let publishers set AI training preferences once across several companies, with differing adoption timelines.
Source details
The Verge · AI · Emma Roth · theverge.comPublished · added here
Original title: The biggest names in AI are backing a new way to opt out of scraping