__usage_ai-training: Google, Microsoft, OpenAI and Anthropic back one robots.txt AI training control. What it does, what it does not, and what to change now

What is __usage_ai-training in robots.txt?

It is a shared "policy token" that goes where a crawler name normally goes in robots.txt. Instead of writing one group for GPTBot, one for ClaudeBot, one for Google-Extended and so on, a site can write a single group addressed to __usage_ai-training and every AI provider that honours the signal reads it as the site’s preference about training generative AI models on its content. The design comes from the Internet Engineering Task Force’s AI Preferences working group (AIPREF) and is written up as an individual Internet-Draft by Kevin Kelley of Anthropic, dated 30 October 2026.

The draft deliberately adds nothing new to robots.txt grammar. In its own words, "It introduces no new directive names and no new grammar: the path language, longest-match precedence, and group structure of robots.txt are not changed." The example in the draft, and in Kelley’s message to the working group list, is three lines: User-Agent: __usage_ai-training, Disallow: /, Allow: /press-kit/. Kelley reads that as "a preference against AI training for everything except /press-kit/, and a preference in favor of it there."

Two rules in the draft matter more than the syntax. First, a wildcard User-agent: * group "MUST NOT be interpreted as matching the policy token", so your existing catch-all rules say nothing about training. Second, if no group names the token, "no preference has been expressed". The draft also states that the token "is not a directive for crawling behavior" and that "Private agreements MAY override preferences expressed via this mechanism". Licensing deals, in other words, sit above robots.txt.

Who announced what on 30 October 2026, and when it takes effect

Four companies moved on the same day. Google’s John Mueller posted on the Search Central blog that "we’re announcing our plans to support a shared AI training control in robots.txt", building on Google-Extended, "the AI training control we’ve offered since 2023". Microsoft’s Krishna Madhavan announced on the Bing Webmaster blog "newer, simpler, and clearer preferences that allow you to make choices about training, grounding, or both while remaining eligible for Bing search results". OpenAI said it will honour the common signal and "aim to implement the signal in the coming months". Anthropic’s Kelley wrote that "Anthropic has already started implementing support for this draft", adding "It’ll take us some time to add support across our services".

Does blocking AI training remove you from AI search answers?

Not by design, but at Google it can in practice. The token is defined against the AIPREF vocabulary’s meaning of AI Training: "Using an asset to modify the learned parameters of a generative AI model." Grounding, which is what happens when ChatGPT, Gemini or Copilot fetch a page at answer time and cite it, is a different use. Microsoft is explicit that the two are separate choices and that "Indexing and ranking pages in traditional search results is not grounding." OpenAI’s post says its existing crawler controls, which separate GPTBot for training from OAI-SearchBot for search, continue to apply.

Google is where you have to read carefully. Mueller’s post says that when a site opts out with the token "and Google-Extended is not explicitly specified", Google "plans to interpret that as if the publisher had disallowed Google-Extended". Google’s crawler documentation defines Google-Extended as the token that controls whether content "may be used for training future generations of Gemini models" and "for grounding" in Gemini Apps and Vertex AI. The same post spells out the consequence: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." So a bare opt-out costs you Gemini grounding unless you add a separate Google-Extended group that allows it.

What it never touches is classic search or Google’s own AI features in Search. Google-Extended is not a ranking signal and does not affect inclusion in Google Search, and AI Overviews and AI Mode are governed by the Search Console "Search generative AI" setting we covered in should you opt out of Google AI Overviews. Microsoft says its new preferences "govern use of content, not crawl access", and that an Allow setting does not override a robots.txt rule that blocks Bingbot.

How many sites block AI crawlers today, and what they get wrong

The reason a shared token is welcome is visible in any large robots.txt sample. AI Crawler Check’s October 2026 study parsed the robots.txt of the Tranco top 10,000 domains on 5 October 2026, with 4,962 valid files as its denominator. Nearly one in five blocks at least one training crawler, and the list of named bots runs to a dozen. The same study found that 347 of the 710 domains blocking GPTBot still allow OAI-SearchBot, which is the distinction between training and search that the new signal is meant to make easy.

Two other samples point the same way. Suff Digital’s HTTP Archive study, published 24 September 2026, counted only rules that name a bot and found 9.9% of 15.5 million sites block at least one AI crawler, rising to 28.9% of the top 1,000. Crawl Readiness tested the top 1,000 on 12 October 2026 and found something robots.txt studies usually miss: 78 of 549 reachable sites, 14.2%, refuse at least one of OAI-SearchBot, PerplexityBot, GPTBot or ClaudeBot at the server while robots.txt allows it. A site can write a perfect robots.txt and still be invisible to AI search because a firewall rule says no.

What practitioners are saying

Fox updated her reference on allowing and blocking LLM crawlers the same day. Her two caveats are the ones to remember: the shared group "adds a preference, it doesn’t replace them", so a company’s own token wins where both apply, and "this group never blocks crawling". Her per-company table also notes that the shared signal would remove a site from Gemini grounding at Google "unless you give Google-Extended its own Allow group".

Hake, whose name appears in the draft’s contributors table alongside engineers from Google, Microsoft, OpenAI and Anthropic, wrote in a follow-up that he remains "quite unhappy about one thing in the current draft", that it exempts preferences from applying to anything a user does, which he says hurts creators dealing with AI copycats, but that "generally the process has been productive, and it’s not over yet". Search Engine Journal’s report makes the same status point: the document is "an individual draft, which does not represent official IETF standardization activity", and none of the companies supports the line today.

Illyes’s point, reported eight days before the announcement, is the frame for all of this. Robots.txt tells a crawler whether it may fetch; it "doesn’t dictate what a system does with the data after it has already crawled it". The new token is the industry’s attempt to bolt a usage preference onto an access protocol without changing the protocol, and whether it works depends entirely on which providers choose to read it.

What to do now: a robots.txt checklist for the next 90 days

Nothing takes effect before 1 February 2027, so this is a planning window, not an emergency. The order below is the one we use with clients in technical SEO and AI search optimisation engagements.

1. Separate the two decisions. "Should models train on our content?" and "Should AI answers cite us?" are different questions with different levers. Most businesses that sell something want the second regardless of the first, because an answer that names you is a referral.
2. Inventory what you block today. List every AI token in robots.txt (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot, Bingbot, Meta-ExternalAgent, CCBot, Bytespider) and what each line actually does. Check whether any group is only reached through User-agent: *.
3. If you want to opt out of training, add the shared group now. Three lines: User-Agent: __usage_ai-training, Disallow: /, plus Allow lines for anything you are happy to have trained on, such as a press kit. It is harmless before support arrives and saves a second edit in 2027.
4. Then protect Gemini grounding explicitly. Add a separate User-agent: Google-Extended group with Allow: / if you want to keep appearing in Gemini Apps answers. Without it, Google plans to treat your opt-out as a Google-Extended disallow, which covers grounding too.
5. If you want to be cited, check you are not blocking the search crawlers. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot need crawl access. Then test at the server: fetch your key pages with those user agents and make sure the firewall or CDN is not returning 403 or 429 while robots.txt says Allow.
6. Audit noarchive and nocache on Bing. Find every page carrying them, decide whether the intent was AI-related, and diary 15 December 2026 and 1 March 2027. Publishing a page-level ai-grounding preference only counts once Bingbot recrawls the page; IndexNow can prompt that but does not guarantee it.
7. Leave the Search Console AI setting where it is unless you have tested. The generative AI control for AI Overviews and AI Mode is a separate switch with its own trade-offs; see our piece on the opt-out before touching it.
8. Write the policy down. The IETF draft lets private agreements override robots.txt. If you license content to any AI provider, your robots.txt should match the contract, not contradict it.

How to measure whether any of this changed your AI visibility

Microsoft says plainly that "Neither choice guarantees inclusion, traffic, or payment", and Google’s generative AI reports in Search Console show impressions rather than clicks. That makes a baseline essential before you change a line. Record, per engine, how often your domain is cited for the queries that matter, which pages carry the citation, and what the referral and assisted conversions look like. Bing Webmaster Tools has an AI performance report and the Topics Portfolio we tested with 972 queries; Search Console has the generative AI performance report; ChatGPT and Gemini referrals need a clean attribution setup, since Gemini stopped adding its UTM tag around 28 September and ChatGPT’s tag went missing from most links between 9 and 16 October. If you would rather have that done for you, our reporting service builds exactly this baseline before any robots.txt or crawl policy change goes live, and our SEO team handles the implementation.

The honest summary is that 30 October 2026 changed the paperwork, not the plumbing. A single token will soon say "do not train on this" to four of the largest AI companies at once, which is progress for publishers and a relief for anyone maintaining a twelve-group robots.txt. It does not decide whether your business shows up when someone asks an AI assistant for a recommendation. That still comes down to being crawlable by the search bots, being the clearest answer on the page, and being measured well enough to know whether it worked.