AI Search Readiness: Common Technical SEO Problems That Prevent Websites from Being Discovered

Where the numbers come from

On 29 September 2026 SitequiryBot crawled 106 well-known websites (mostly Slovenian and German, plus English and other languages; shops, news, public services, companies, tourism, NGOs). The crawl for 90 sites completed; 13 answered our crawler with 403, 429 or a bot challenge and 3 redirected to another domain. For each completed site we analyzed up to 20 pages and checked the 12,381 internal links found on them, using the same checks as the paid Site Report.

Every figure below counts websites, not pages: a site counts once for a problem whether one URL or fifty were affected. The sample is a convenience sample of well-known sites, not the web as a whole, and 20 pages are not a full crawl, so a site that "passed" may still have the problem elsewhere. The robots.txt, sitemap, llms.txt and markup figures are published with their method in our AI search readiness study; the technical figures on this page come from the same crawl and are available as CSV and JSON under CC BY 4.0. No per-site results are published.

Classic SEO and AI visibility: same foundation, different last step

Google is explicit that its AI features need nothing extra: its page on AI features and your website says a page must be indexed and eligible to appear with a snippet, and that the usual SEO fundamentals apply. There is no special markup or file that qualifies a page for AI Overviews or AI Mode.

The other AI search products send their own crawlers. OpenAI uses OAI-SearchBot to surface sites in ChatGPT search, separately from GPTBot, which collects training data. Anthropic documents Claude-SearchBot, Claude-User and ClaudeBot and says they honor robots.txt; Perplexity uses PerplexityBot for its search results. Whatever these systems do later, they start where a search engine starts: fetch the URL, follow links, read the HTML.

The last step differs. A search engine ranks a list of pages; an AI answer selects a few passages and may cite them. None of these companies publishes how sources are chosen for a citation, and no crawl, ours included, can measure it. What the technical layer decides is narrower and easier to check: whether a page can be in the pool at all.

1. Crawlers are turned away before they see a page

The first gate is robots.txt. In our study, 14 of the 90 sites (16%) blocked at least one of the four AI search crawlers we test, and 18 (20%) blocked at least one of 15 AI crawlers of any kind. Whether a block is deliberate cannot be seen from outside, so check that a rule written to stop AI training does not also name a search crawler. The AI crawlers and robots.txt guide explains which user agent does what.

The second gate is the firewall or CDN, and robots.txt says nothing about it. 13 of the 106 sites in our sample did not let our crawler in at all, answering with 403, 429 or a challenge page. We found the same pattern on our own site: on 13 October 2026, sitequiry.com allowed every crawler in robots.txt, yet the hosting firewall returned 403 to the Bytespider user agent. Bot protection that cannot tell a search crawler from a scraper can lock out OAI-SearchBot or PerplexityBot as easily as anyone else, and the owner never sees it in robots.txt.

A third, quieter gate is noindex. Google points out that a page blocked by robots.txt cannot show its noindex tag, because the crawler never fetches it (Google: block indexing). In our crawl one homepage carried noindex: one site, but its entire front door was asking to be left out.

• Fetch your robots.txt and read every User-agent group that names an AI crawler; the AI crawler checker does this for 15 of them.
• Ask your host or CDN which bots their firewall challenges. Cloudflare, for example, has a setting that blocks AI crawlers whatever robots.txt says.
• Check that pages you want found carry no noindex in the HTML or in an X-Robots-Tag header.

2. Internal links that lead nowhere, or the long way round

Crawlers discover most pages by following links. Of the 12,381 internal links we checked, 64 led to an error page (60 of them 404s). That sounds small, but they were spread across 18 of the 90 sites (20%), and on 10 of those 18 sites the same broken target was linked from three or more of the analyzed pages, which usually points to a menu, footer or template link.

Redirects were far more common: 67 sites (74%) linked internally to at least one URL that redirected, mostly from http:// to https:// or to a version with or without a trailing slash. A single hop is harmless. Chains are the problem: 23 sites (26%) had at least one URL that needed two or more redirects. Google's crawlers follow up to 10 hops (Google: HTTP status codes), and its site-move guidance asks for a redirect straight to the final URL, or a chain of fewer than five hops and ideally no more than three (Google: site moves). The other AI crawlers do not document their limits, so the safe assumption is that every extra hop is another request that may not be made.

Both problems have the same fix: update the link itself to the final URL, in the template that prints it, rather than adding another redirect. The step-by-step fixes for audit findings cover broken links and redirect chains with a re-test for each.

3. Sitemaps that contradict the pages they list

A sitemap is the list of URLs you ask search engines to crawl and index. 75 of the 90 sites had one. Among those, 15 (20%) listed at least one URL that redirected or returned an error, and 6 (8%) listed a page that carried noindex. Each is a contradiction: the sitemap says "index this" while the page says "I have moved", "I do not exist" or "do not index me".

Google asks for the canonical URLs you want in search results and uses lastmod only when it is consistently accurate (Google: build a sitemap). Bing makes the same point with AI search in mind: in a July 2025 post it recommends a complete sitemap with a truthful lastmod and IndexNow for changed pages so that AI-powered results stay fresh.

• Generate the sitemap from the same data that decides canonical and noindex, so a page that is redirected or excluded leaves the sitemap automatically.
• Set lastmod to the date the content last changed, not the date the file was built.
• Declare the sitemap in robots.txt. 70 of the 84 sites with a readable robots.txt did; the rest rely on search engines finding it at the default address.

4. Canonical, duplicate and language signals that disagree

A canonical tag tells search engines which of several similar URLs is the main one. Google calls it a strong signal, alongside redirects, while listing a URL in the sitemap is a weak one (Google: canonical URLs). A canonical that points at a page which redirects, errors or is noindex sends crawlers to a dead end. We found this on only one site. Canonical errors were rare in this sample, but a wrong canonical in a template affects every page built from it.

Duplicate titles were more common: 16 sites (18%) had at least two indexable pages with the same title. When two pages present themselves identically, a search engine has to work out on its own which one answers a query; Google's title link guidelines ask for a distinct, descriptive title on every page.

Multilingual sites add hreflang, which only works when every language version lists itself and all others and each link is returned; otherwise the annotations may be ignored (Google: localized versions). Of the 31 sites with hreflang, 4 (13%) broke that loop on at least one page. The canonical and hreflang checker tests one URL and its alternates in a few seconds.

5. Content that exists only after JavaScript runs

On 5 of the 90 sites (6%) at least one analyzed page was a client-rendered shell: the HTML the server sent contained no readable content until JavaScript ran. Three more sites were left out of the study's page figures for the same reason, because almost none of their pages had content in the HTML.

Google renders JavaScript, but in a separate step after crawling, and still recommends server-side rendering or pre-rendering, partly because not every crawler can run scripts (Google: JavaScript SEO basics). For AI crawlers the best public measurement is Vercel's analysis of December 2024: none of the major AI crawlers rendered JavaScript at that time; OpenAI's and Anthropic's crawlers fetched script files but did not execute them, while Gemini (through Google's infrastructure) and Applebot did render. That may change, but today a page whose text appears only in the browser is likely invisible to some AI search crawlers.

The test is simple: open the page source (not the inspector) and search for a sentence from the main text. If it is not there, ask your developer about server-side rendering or static generation for the pages that matter.

6. Pages that are hard to understand once they are read

Being fetched is not the same as being understood. The study found that 59% of homepages carried some schema.org markup but only 35% named the organization behind the site, and on 43% of the sites checked two or more indexable pages had no H1. Structured data does not earn a place in AI answers on its own; Google's rule is that it must describe what is visible on the page (Google: structured data guidelines), and its May 2025 advice on AI experiences puts unique, useful content first, before any technical point.

In practice that means one clear H1 that names the topic, a title and description that differ from every other page, an Organization block that matches the name and address on the page, and text that answers the question the page is about in plain sentences. The structured data guide has a minimal Organization example.

What this crawl cannot tell you

• Whether a site is cited in ChatGPT, Perplexity, Claude or Google's AI answers. We measured readiness signals only; no figure here is a visibility or ranking measurement.
• Whether any of these problems caused a ranking change. They are documented obstacles to crawling and indexing, not proven AI citation factors.
• Anything beyond 20 pages per site, or what a crawler from another location or with another user agent would have received.
• Speed and Core Web Vitals, which were not measured for this sample.

A checking order for your own site

• Access: robots.txt groups for search and AI crawlers, plus firewall or CDN bot settings. Use the AI crawler checker.
• Indexability: no unwanted noindex, canonicals that point to live, indexable pages. Use the meta tag checker.
• Links: no internal links to 404s, no redirect chains; update links to their final URL.
• Sitemap: only canonical, indexable URLs that answer 200, with a truthful lastmod.
• Rendering: the main text is in the page source.
• Languages: hreflang lists every version, including the page itself, and every link is returned. Use the canonical and hreflang checker.
• Understanding: one H1, unique title and description, structured data that matches the page.

The free analysis checks one URL for these signals, plus speed through Google PageSpeed. The paid Site Report runs the same checks as this crawl on up to 20 pages of a site and checks the internal links it finds. When the fixes need a developer, the GE-KO team behind Sitequiry can take on the implementation.

Frequently asked questions