Three kinds of AI crawlers
Training crawlers collect public pages to train future models. Blocking them does not remove you from AI search results.
Search and retrieval crawlers build the index an assistant uses to find and cite sources. Blocking them is what makes you invisible in AI answers.
User-triggered fetchers load one page because a person asked the assistant to read it. Vendors treat these differently from automated crawling, and some state that robots.txt rules may not apply to them.
The user-agent names that matter
- OpenAI:
GPTBot(training),OAI-SearchBot(ChatGPT search),ChatGPT-User(user requests). - Anthropic:
ClaudeBot(training),Claude-SearchBot(search),Claude-User(user requests). - Perplexity:
PerplexityBot(search index),Perplexity-User(user requests). - Google:
Google-Extendedis a control token, not a separate crawler. It decides whether content fetched by Googlebot may be used for Gemini models and for grounding answers in Gemini Apps. It does not affect Google Search, AI Overviews or AI Mode. - Apple:
Applebot-Extendedcontrols use for Apple's AI training;Applebotitself powers Siri and Spotlight. - Others you will see in logs:
CCBot(Common Crawl, widely used for training),meta-externalagent(Meta),Amazonbot,Bytespider(ByteDance),DuckAssistBot(DuckDuckGo).
A sensible default robots.txt
For most business sites the goal is to be found and cited. A common choice is to allow search and user-triggered bots and decide separately about training. This example allows AI search while opting out of training:
# Search engines and AI search: allowed
User-agent: *
Allow: /
# AI training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlIf you are happy for your content to be used in training, which can help models know your brand, simply leave the second block out. Either way, never disallow OAI-SearchBot, Claude-SearchBot or PerplexityBot if you want to appear in those assistants.
Blocking Google-Extended also stops Gemini Apps from grounding answers in your pages, so leave it out of the block if being cited in Gemini Apps matters to you.
New since 30 October 2026: one training preference for all providers
Instead of listing every training bot, an IETF draft proposes one group that states your training preference to every provider that supports it. Google and Microsoft plan to read it from 2027, OpenAI and Anthropic have announced support, and no provider applies it yet:
User-agent: __usage_ai-training
Disallow: /It is a preference, not a crawler block, and User-agent: * does not count as one. Keep the vendor groups above until the providers you care about read the shared signal. Our guide to AI training preferences explains the rules, Google's reading of it for Google-Extended and Bing's page-level settings.
Check your CDN and firewall too
robots.txt is only a request. The more common problem in practice is the opposite: a firewall or CDN that blocks AI crawlers before they ever read robots.txt. Cloudflare, for example, started blocking known AI crawlers by default for new domains in July 2025, and many security plugins have similar switches.
If your robots.txt allows a bot but the site still never appears in AI answers, check the bot settings of your CDN, hosting firewall and security plugins, and look for 403 responses to these user agents in your server logs.
What Sitequiry checks
The analysis reads your robots.txt and reports which AI crawlers are allowed or blocked, separating training from search bots, and flags when a blanket rule blocks everything. It also shows your shared AI training preference and your Bing page-level values as information, without changing a score. It checks access rules, not whether a CDN blocks the bot at network level, because that can only be seen from the bot's own IP ranges.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and pages a user asks about are fetched by ChatGPT-User.
Does Google-Extended affect AI Overviews?
No. AI Overviews and AI Mode use the normal Google Search index crawled by Googlebot. To limit how your content appears there, use preview controls such as nosnippet or max-snippet, which also affect regular snippets.
Is robots.txt legally binding?
No. It is a widely respected convention. Major AI companies document that their crawlers follow it, but it does not technically prevent access.
How often should I review this?
At least twice a year. New crawlers appear regularly and vendors rename or split existing ones.
Check your own robots.txt
Enter your domain to see which AI crawlers your robots.txt allows or blocks, and which line decides.
Check your own site in about a minute
The free analysis measures the points from this guide on your URL and shows what to fix first.