AI Training Preferences: AIPREF, robots.txt and AI Search Visibility

What changed on 30 October 2026

Kevin Kelley of Anthropic published the Internet-Draft draft-kelley-aipref-training-preference-signal-00 in the IETF AI Preferences (AIPREF) work. It is an individual draft: the IETF datatracker states that it has no formal standing and is not endorsed by the IETF, and it expires on 3 May 2027 unless it is renewed or adopted.

On the same day Google, Microsoft and OpenAI announced that they plan to support it, and Anthropic wrote on the IETF mailing list that it has started implementing it. Before this, every provider had its own training token, so a site that wanted to opt out had to list GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and others one by one.

How the shared signal works

The signal is an ordinary robots.txt group whose user-agent line names a policy token instead of a crawler. Disallowed paths state a preference against AI training, allowed paths a preference in favour. The rules that matter:

• It is not a crawler. No bot identifies itself as __usage_ai-training, and the group blocks no visit. To limit crawling, use the crawler's own product token.
• User-agent: * never counts. The draft says a wildcard group must not be treated as matching the policy token, so User-agent: * with Disallow: / is a crawl block for everyone, not a training preference.
• No group, no preference. A robots.txt without the token states nothing. That is neither an opt-out nor a permission; each provider then applies its own rules.
• Matching follows RFC 9309: the token matches regardless of upper or lower case, consecutive User-agent lines share one group, and the longest matching path rule decides, with Allow winning a tie.
• Specific beats general. The draft says more specific expressions should take precedence, and providers may document their own tokens as more specific. A GPTBot or Google-Extended group of your own can therefore override the shared preference for that provider.
• Private agreements, such as a licensing deal with an AI company, may override what robots.txt says.

User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

This example opts the whole site out of training except the press kit: /press-kit/ is longer than /, so it decides for every URL below it.

Crawling, training and grounding are three different questions

Most robots.txt mistakes with AI come from mixing these up. The shared signal answers only the second question.

What each provider announced

None of the four supports the signal yet. Their statements differ in timing and in how the shared token interacts with their own controls.

We found no announcement from Apple, Meta, Amazon, Perplexity, ByteDance or Common Crawl as of 31 October 2026. Their own tokens (Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider) remain the only way to address them in robots.txt.

Google-Extended: the trade-off between training and Gemini Apps

Google-Extended is not a crawler. It tells Google whether pages Googlebot fetched may be used to train Gemini models and to ground answers in Gemini Apps and in Grounding with Google Search on Vertex AI. Google's crawler documentation says it does not affect inclusion or ranking in Google Search.

The two uses come as a pair. If you block Google-Extended, you opt out of Gemini training and Gemini Apps grounding together; if you allow it, you allow both. Once Google reads the shared token, a training opt-out without a Google-Extended group will have the same effect as blocking Google-Extended. If you want Gemini Apps to keep using your pages, add your own group:

User-agent: __usage_ai-training
Disallow: /

# Google: keep Gemini Apps grounding (this also allows Gemini training)
User-agent: Google-Extended
Allow: /

AI Overviews and AI Mode are part of Google Search and use the Googlebot index. To limit them, use nosnippet, max-snippet or data-nosnippet, which also affect normal snippets.

Bing: page-level training and grounding

Microsoft reads the shared token in robots.txt for training and adds two page-level preferences that apply to Bingbot. There is no site-wide grounding rule in robots.txt, and a page-level training value overrides robots.txt for that page.

• allow is the default for both. disallow for training excludes the page from training generative models; for grounding it stops Microsoft from using the page to inform generated answers.
• disallowextended exists for grounding only. It keeps grounding in Microsoft's own experiences (Copilot, Bing's AI Mode and AI Overview, Microsoft 365) and excludes third-party grounding services built on Bing.
• 15 December 2026: sites that use NoArchive or NoCache can record a temporary grounding preference in Bing Webmaster Tools.
• 1 February 2027: the new preferences take effect as pages are crawled. Where they are found on a site, NoArchive and NoCache on that site are no longer read as AI preferences.
• 1 March 2027: NoArchive and NoCache stop counting as training or grounding preferences altogether.
• 30 September 2027: temporary grounding preferences expire, or earlier when a recrawl finds the page's own ai-grounding value.
• None of this changes Bingbot's crawl access or eligibility for Bing search.

<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallowextended">

X-Robots-Tag: Bingbot: ai-grounding:disallow

Common mistakes

• Using User-agent: * with Disallow: / as an AI opt-out. It blocks Googlebot, Bingbot and every other crawler without its own group, and it still is not a training preference.
• Misspelling the token. __usage-ai-training, _usage_ai-training or usage_ai-training match nothing, so the file states no preference.
• Putting the token in the same group as a search crawler. User-agent: Googlebot directly followed by User-agent: __usage_ai-training and Disallow: / blocks Googlebot from the whole site.
• Expecting a training opt-out to remove you from ChatGPT search, Claude or Perplexity answers. Those use their search bots; only blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot does that.
• Opting out of training through the shared token and assuming Gemini Apps will still cite you. Without your own Google-Extended group, Google plans to read it as a Google-Extended block.
• Leaving contradicting groups in place, such as a shared opt-out next to an old User-agent: GPTBot with Allow: /. The specific group is likely to win for that provider.
• Relying on NoArchive or NoCache to limit Bing's AI use after 1 March 2027.

What it means for SEO, AEO and GEO

For classic SEO the effect is nil by the providers' own statements: Google says Google-Extended does not affect Search, and Microsoft says Bing search eligibility is unchanged. A training opt-out is a business decision about your content, not a ranking factor, so Sitequiry does not lower any score when a site has no opt-out, or when it has one.

For visibility in AI answers it depends on the provider. Answers in ChatGPT search, Claude and Perplexity depend on their search bots, which the training signal does not touch. Gemini Apps depend on Google-Extended, which the shared opt-out will switch off unless you add your own group. Bing's AI answers depend on the page-level grounding value, which is separate from training.

Check your configuration

The free AI crawler checker reads your robots.txt and homepage and shows crawler access, your shared training preference with the deciding lines, your Google-Extended, GPTBot and ClaudeBot groups, your homepage's Bing values, and warnings for the conflicts above. Its generator writes the __usage_ai-training group with path exceptions, without touching the rules for Googlebot or Bingbot.

Re-check in early 2027, when Google and Microsoft start reading the signal, and before 1 March 2027 if you rely on NoArchive or NoCache for Bing.

Frequently asked questions