One Switch to Stop AI Training Scraping: Google, OpenAI, Microsoft and Anthropic Back a Single Robots.txt Signal

One rule instead of four

Google, Anthropic, OpenAI and Microsoft have committed to honouring a new robots.txt directive that lets publishers state their preferences for AI training scraping in one place, rather than writing a separate rule for each company Engadget. The companies have adopted a proposal under consideration at the Internet Engineering Task Force (IETF), the internet’s open standards body, written by Anthropic technical lead for standards Kevin Kelley Engadget.

The change matters because of how messy the current system is. As OpenAI and Google explained in their announcements, website owners currently have to identify individual crawlers and add separate rules for each provider Engadget. Every new AI company with its own crawler adds another line to maintain, another name to track, and another chance for a publisher’s preference to be missed. A single shared token removes that friction at a stroke — at least for the four companies that have signed up.

How the new signal works

Publishers set up a new user-agent section in their robots.txt file specifically for AI training, which looks like this: User-Agent: __usage_ai-training, followed by Disallow: / and optional Allow lines for specific paths such as a press kit Engadget. That blocks crawlers specifically meant to gather data for AI training, while the Allow section gives site operators the option to permit training use for selected parts of a site Engadget.

The limits are just as important as the promise. This new user-agent section does not prevent crawling altogether or prevent content from showing up on Google Search Engadget. Google will interpret it specifically as a stop sign for its Google-Extended user agent, which crawls websites for materials used to train Gemini models Engadget. And it cannot undo any past scraping: if a company has already crawled a page, it already has the content, and the new signal only prevents future training use Engadget.

A staggered rollout, not a switch-on

The four companies are not moving at the same speed. Anthropic has already started supporting the new directive, while Microsoft said it will start supporting it on 1 February 2027, Google said it will support it in early 2027, and OpenAI pledged to implement the new signal in the coming months Engadget. The proposal itself is still only under consideration at the IETF, not a finished standard, and reporting on the announcement noted that the solution will not fully take effect for months Malay Mail.

That staggered timetable is the honest caveat on an otherwise genuinely useful step. For years, publishers’ central complaint has been that opting out meant negotiating with each AI lab separately, while legal battles over training data ground on in the background. A shared signal does not settle who owns training data or what past scraping was worth, but it does give every website — from a global newspaper to a small independent blog — one simple, machine-readable way to say no to future training. The test now is whether the remaining AI companies follow, and whether the promise survives contact with each firm’s own crawler infrastructure.

What to watch

Watch for the IETF proposal’s progress, for Meta and other AI firms deciding whether to honour the same token, and for the first publisher reports of whether the signal is actually respected once Microsoft’s February 2027 start date arrives. For website owners, the practical move is simple: add the section now, and check again when each company’s stated start date passes.