Test whether robots.txt allows a URL
Enter a public URL and choose a crawler. The tester reads the robots.txt of that site, shows whether the crawler may fetch the URL and highlights the line that decides it. No account required.
How a crawler picks its rules
A crawler follows the group that names its own user-agent token; only when no group names it does User-agent: * apply. Two groups that name the same crawler are combined, but a named group and the * group are not (Google Search Central). Inside the group the most specific rule (the longest path) wins; when an Allow and a Disallow match equally, Google uses the less restrictive one. Paths are case-sensitive, user-agent names are not.
An example from nytimes.com
On 10 November 2026 the robots.txt of nytimes.com listed User-agent: * and User-agent: Googlebot together on lines 15 and 16, and named Googlebot again in a second group on line 130. Googlebot obeys both of its groups combined and nothing else; GPTBot has its own group with Disallow: / on lines 232 and 233, so it is blocked from the whole site. The tester shows which group and line a crawler ends up with in cases like this.
Blocking crawling is not blocking indexing
robots.txt controls which URLs a crawler may fetch, not which ones appear in search. Google can still index a disallowed URL that is linked from elsewhere, shown without a description (Google Search Central). To keep a page out of results, use noindex and leave the page crawlable, otherwise Google never sees the noindex (Google Search Central). A noindex line inside robots.txt has had no effect at Google since 1 September 2019 (Google Search Central blog).
When robots.txt does not answer normally
Google treats a missing robots.txt (any 4xx answer except 429) as "no restrictions". A server error (5xx) is different: Google stops crawling the site for 12 hours and then uses its last good copy for up to 30 days (Google Search Central). Google reads only the first 500 KiB of the file. To see what AI crawlers in particular may do across your whole site, use the AI crawler checker.
Frequently asked questions
Does Disallow remove a page from Google?
No. Disallow only stops crawling. A disallowed page can still be listed if other pages link to it. Use a noindex meta tag or header on a crawlable page instead.
Why does Googlebot ignore my "User-agent: *" rules?
Because your robots.txt also has a group that names Googlebot. A crawler with its own group follows only that group, so rules from * must be repeated there if they should apply.
Does Google support Crawl-delay?
No. Google ignores crawl-delay lines. Other crawlers may read it, so it can stay in the file, but it does not change how fast Googlebot crawls.
Check the pages crawlers are allowed to see
The free analysis checks a page for SEO, speed with Google PageSpeed, AI answer readiness and AI citability.
Free SEO tools
-
Meta tag checker
Title, meta description, H1 and canonical of one page, with a snippet preview.
-
Canonical & hreflang checker
Canonical URL, hreflang codes and return links between language versions.
-
Redirect checker
Every redirect hop of one URL, the http/https and www variants, and the SEO and security headers of the final answer.
-
Schema checker
Reads every JSON-LD block of a page: syntax errors, schema.org types, and the properties Google documents for Organization, LocalBusiness, Product, Article, BreadcrumbList, Event and Recipe.
-
Open Graph checker
Lists the Open Graph and twitter: tags of a page, checks that the share image loads and how large it is, and shows an illustrative share card.
-
Sitemap checker
Finds the XML sitemap of a site, checks that it parses, counts its URLs, compares it with the protocol limits and requests a sample of the listed URLs.
-
AI crawler checker
Which AI crawlers your robots.txt allows or blocks, and whether llms.txt exists.
Related guides
-
AI crawlers and robots.txt: which bots to allow in 2026
Training bots, AI search bots and user-triggered fetchers are different things. Here is how to allow the ones that bring visibility.
-
Why Is My Website Not Showing Up on Google?
Three different problems hide behind one complaint. How to tell which one you have, starting from what Google already reports.
-
How to Fix SEO Audit Issues Without Breaking Your Website
Ten audit findings, each with the symptom, how to confirm it, a safe fix, how to re-test and what to watch out for.