Three kinds of AI crawlers
Training crawlers collect public pages to train future models. Blocking them does not remove you from AI search results.
Search and retrieval crawlers build the index an assistant uses to find and cite sources. Blocking them is what makes you invisible in AI answers.
User-triggered fetchers load one page because a person asked the assistant to read it. Vendors treat these differently from automated crawling, and some state that robots.txt rules may not apply to them.
The user-agent names that matter
- OpenAI:
GPTBot(training),OAI-SearchBot(ChatGPT search),ChatGPT-User(user requests). - Anthropic:
ClaudeBot(training),Claude-SearchBot(search),Claude-User(user requests). - Perplexity:
PerplexityBot(search index),Perplexity-User(user requests). - Google:
Google-Extendedis a control token, not a separate crawler. It decides whether content fetched by Googlebot may be used for Gemini models and grounding. It does not affect Google Search, AI Overviews or AI Mode. - Apple:
Applebot-Extendedcontrols use for Apple's AI training;Applebotitself powers Siri and Spotlight. - Others you will see in logs:
CCBot(Common Crawl, widely used for training),meta-externalagent(Meta),Amazonbot,Bytespider(ByteDance),DuckAssistBot(DuckDuckGo).
A sensible default robots.txt
For most business sites the goal is to be found and cited. A common choice is to allow search and user-triggered bots and decide separately about training. This example allows AI search while opting out of training:
# Search engines and AI search: allowed
User-agent: *
Allow: /
# AI training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlIf you are happy for your content to be used in training, which can help models know your brand, simply leave the second block out. Either way, never disallow OAI-SearchBot, Claude-SearchBot or PerplexityBot if you want to appear in those assistants.
Check your CDN and firewall too
robots.txt is only a request. The more common problem in practice is the opposite: a firewall or CDN that blocks AI crawlers before they ever read robots.txt. Cloudflare, for example, started blocking known AI crawlers by default for new domains in July 2025, and many security plugins have similar switches.
If your robots.txt allows a bot but the site still never appears in AI answers, check the bot settings of your CDN, hosting firewall and security plugins, and look for 403 responses to these user agents in your server logs.
What Sitequiry checks
The analysis reads your robots.txt and reports which AI crawlers are allowed or blocked, separating training from search bots, and flags when a blanket rule blocks everything. It checks access rules, not whether a CDN blocks the bot at network level, because that can only be seen from the bot's own IP ranges.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and pages a user asks about are fetched by ChatGPT-User.
Does Google-Extended affect AI Overviews?
No. AI Overviews and AI Mode use the normal Google Search index crawled by Googlebot. To limit how your content appears there, use preview controls such as nosnippet or max-snippet, which also affect regular snippets.
Is robots.txt legally binding?
No. It is a widely respected convention. Major AI companies document that their crawlers follow it, but it does not technically prevent access.
How often should I review this?
At least twice a year. New crawlers appear regularly and vendors rename or split existing ones.
Check your own site in about a minute
The free analysis measures the points from this guide on your URL and shows what to fix first.