One robots.txt Token Now Covers AI Training Opt-Outs
Google, Microsoft, OpenAI and Anthropic backed a single robots.txt policy token on October 30. How it works, who ships when, and what not to delete.
On October 30, 2026, Google, Microsoft, OpenAI and Anthropic lined up behind a single shared robots.txt signal for AI training preferences — one token instead of one rule per vendor.
For years, opting out of AI training meant maintaining a growing list of crawler names, one at a time, with new ones appearing as new products shipped. The shared signal replaces that patchwork with a single policy token every participating provider agrees to read the same way.
How the token works
The mechanism is deliberately boring. It introduces one policy token, __usage_ai-training, that sits in the User-Agent line of an ordinary robots.txt group. There are no new directives and no new grammar: the path language, longest-match precedence and group structure of RFC 9309 are unchanged.
The draft's own example expresses a preference against training for all paths except /press-kit/:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Two compatibility details decide whether your file actually says anything. First, parsers that do not recognise the token simply ignore the group, the same way they ignore rules for any bot they have never heard of — so adding it should not disturb how other crawlers read the rest of the file. Second, and more important: wildcard groups do not count. The draft says a User-Agent: * group "must not be interpreted as matching the policy token." If your robots.txt has no __usage_ai-training group, you have expressed no training preference at all, whatever your catch-all rules say.
The signal is defined in an IETF Internet-Draft from the AI Preferences (aipref) working group, "Machine-Readable Training Preference Signal" (draft-kelley-aipref-training-preference-signal-00), authored by Kevin Kelley of Anthropic, dated 30 October 2026 and expiring 3 May 2027. The Verge describes Kelley as a technical lead for standards at Anthropic.
Who committed to what, and when
The timelines are uneven, and that is the detail most coverage skips. There is no single switch-on date.
| Provider | Commitment |
|---|---|
| Aims to support the control in 2027. A shared-token opt-out that does not mention Google-Extended will be read as disallowing Google-Extended. | |
| OpenAI | Will honor the signal, aiming to implement "in the coming months." No firmer date given. |
| Anthropic | Authored the draft; same-day reporting says implementation has started — reported, not independently confirmed. |
| Microsoft | Says it will adopt the shared standard; no implementation date given. |
Specificity still wins. The draft lets providers treat their own tokens, such as Google-Extended, as more specific than the shared one, and Google confirmed it will work that way: sites that explicitly allow Google-Extended stay eligible for grounding citations in Gemini Apps even if they opt out broadly with the shared signal.
What not to do today
Do not delete your existing GPTBot or Google-Extended rules. OpenAI says the new signal will not replace the controls it already provides through robots.txt, which continue to apply and be respected, and Google says the same about Google-Extended. Those are the controls that work today; the shared token is a draft that is still rolling out.
It is worth being precise about what this is. As the source analysis puts it, this is not a switch that turns AI training off everywhere tomorrow — the timelines are uneven, the specification is still a draft, and a robots.txt preference is not a security lock.
The useful move this week is additive: add the __usage_ai-training group alongside what you already have, set it per directory if your answer differs for product pages and premium guides, and watch for OpenAI's promised implementation update.
More from DangMua