Skip to content
Sitequiry
Free tool

Test whether robots.txt allows a URL

Enter a public URL and choose a crawler. The tester reads the robots.txt of that site, shows whether the crawler may fetch the URL and highlights the line that decides it. No account required.

Reads the robots.txt of the URL's own host, protocol and port. Rules follow Google's robots.txt specification.

How a crawler picks its rules

A crawler follows the group that names its own user-agent token; only when no group names it does User-agent: * apply. Two groups that name the same crawler are combined, but a named group and the * group are not (Google Search Central). Inside the group the most specific rule (the longest path) wins; when an Allow and a Disallow match equally, Google uses the less restrictive one. Paths are case-sensitive, user-agent names are not.

An example from nytimes.com

On 10 November 2026 the robots.txt of nytimes.com listed User-agent: * and User-agent: Googlebot together on lines 15 and 16, and named Googlebot again in a second group on line 130. Googlebot obeys both of its groups combined and nothing else; GPTBot has its own group with Disallow: / on lines 232 and 233, so it is blocked from the whole site. The tester shows which group and line a crawler ends up with in cases like this.

Blocking crawling is not blocking indexing

robots.txt controls which URLs a crawler may fetch, not which ones appear in search. Google can still index a disallowed URL that is linked from elsewhere, shown without a description (Google Search Central). To keep a page out of results, use noindex and leave the page crawlable, otherwise Google never sees the noindex (Google Search Central). A noindex line inside robots.txt has had no effect at Google since 1 September 2019 (Google Search Central blog).

When robots.txt does not answer normally

Google treats a missing robots.txt (any 4xx answer except 429) as "no restrictions". A server error (5xx) is different: Google stops crawling the site for 12 hours and then uses its last good copy for up to 30 days (Google Search Central). Google reads only the first 500 KiB of the file. To see what AI crawlers in particular may do across your whole site, use the AI crawler checker.

Frequently asked questions

Does Disallow remove a page from Google?

No. Disallow only stops crawling. A disallowed page can still be listed if other pages link to it. Use a noindex meta tag or header on a crawlable page instead.

Why does Googlebot ignore my "User-agent: *" rules?

Because your robots.txt also has a group that names Googlebot. A crawler with its own group follows only that group, so rules from * must be repeated there if they should apply.

Does Google support Crawl-delay?

No. Google ignores crawl-delay lines. Other crawlers may read it, so it can stay in the file, but it does not change how fast Googlebot crawls.

Check the pages crawlers are allowed to see

The free analysis checks a page for SEO, speed with Google PageSpeed, AI answer readiness and AI citability.