How ready are websites for AI search?

Who blocks AI crawlers?

We checked whether robots.txt disallows the homepage for 15 AI crawlers, grouped by purpose: 8 training crawlers (they collect pages to train models), 4 AI search crawlers (they build an index an assistant can cite from) and 3 user-triggered fetchers (they load a page because a person asked an assistant about it). Two of the training entries, Google-Extended and Applebot-Extended, are control tokens rather than crawlers. The guide to AI crawlers and robots.txt explains the difference.

training AI search user-triggered

The share of sites that block a given crawler ranges from 6% (for example OAI-SearchBot) to 19% (Bytespider). Googlebot is blocked by 0 of 90 sites.

Blocking a training crawler is a legitimate choice and does not remove a site from AI search. Blocking an AI search crawler can keep a site out of the index an assistant cites from, and 16% of the sites block at least one. We did not measure whether any of these sites appears in AI answers.

On 2 October 2026, we fetched robots.txt again for the 18 sites flagged as blocking and read it with a second, simpler parser. The blocked crawlers matched for 18 of 18. We did not recheck sites that were not flagged.

Readiness signals beyond crawler access

We also looked for these readiness signals: an llms.txt file (a proposed Markdown summary of a site; no major search engine has committed to using it, see the llms.txt guide), a sitemap declared in robots.txt, schema.org structured data and hreflang annotations. None of them guarantees that an AI answer will cite a site.

We counted structured data by presence and did not validate it (see the structured data guide). FAQ markup is optional, so a missing FAQPage is not an error by itself. The hreflang annotation matters only for multilingual sites.

On 2 October 2026, we fetched the 12 detected llms.txt files again. Of these, 12 are real text files, not HTML error pages.

On-page basics that search and AI both read

Search engines and AI systems both read basic page elements such as the title, description, headings and image alt text. The charts show how many sites have gaps in them, using the definitions of the Sitequiry Site Report (see the methodology).

Across 83 sites, we checked 71,406 images. 8,991 of them (13%) have no alt attribute at all. An empty alt="" is correct for decorative images, so we did not count it.

By language group

We show a figure for a language group only if it is based on at least 15 sites; otherwise the cell reads "n too small". The language is the main language of the site as we classified it. The sample has 36 Slovenian, 26 German (mainly Germany and Austria), 19 English and 25 sites in other languages (Croatian, Italian, French, Spanish, Czech, Dutch). These counts include sites that could not be analyzed.

Differences between groups can say more about which sites we picked than about a language or a country, and the groups are small. Read the table as a description, not as a ranking.

Methodology

This is a convenience sample, not a random one. We chose 106 well-known public websites to cover different types and languages: 31 shops and marketplaces, 22 news and magazine sites, 25 public bodies, health and education sites, 15 companies, SaaS and agencies, 8 tourism and hospitality sites and 5 non-profits. Of these sites, 10 are known to use bot protection.

• Crawl. On 29 September 2026, SitequiryBot, the crawler of the Sitequiry Site Report, read each site: the homepage plus further pages, up to 20 pages in total, picked from the sitemap and the main navigation. These pages are not a random sample of a site. The crawler respects robots.txt and pauses between requests. Software: Site Report build 98996b7.
• Completed reports. We completed 90 of 106 reports. We left out 13 sites that blocked the first request (for example with HTTP 403 or 429) and 3 that redirected to another domain.
• Blocking AI crawlers. We count a crawler as blocked when robots.txt disallows the homepage ("/") for it, either in a group that names the crawler or in the "*" group when the crawler has no group of its own. Matching follows RFC 9309. A missing or unreadable robots.txt counts as "no rules", and rules for sections of a site are not counted.
• Pages. Page-level figures describe a site only when at least 5 of its pages could be analyzed. Of the completed reports, 7 had fewer analyzable pages, because of client-rendered pages without content in the HTML, bot protection after the first request, or very little crawlable content. A URL that redirects to another page is not counted twice. We judge title, description, H1 and Open Graph on indexable pages only (no noindex, canonical pointing to the page itself). We do not judge markup and headings on pages whose HTML holds no content because a script renders it.
• Images. An image counts as missing alt text only when its img element has no alt attribute. We ignore tracking pixels and hidden images.
• Shares. Every share is the number of sites with the property divided by the number of sites in the population named in the dataset. The dataset lists both numbers for every figure.

Limits of this study

• Not representative. We chose the sites ourselves, mostly well-known organizations. The study says little about small business websites.
• Sites that blocked our request are not in the figures, so the results lean toward sites that allow automated checks.
• robots.txt is a request, not enforcement. A site may also block crawlers with a firewall or CDN, and may serve different rules to different crawlers. We read robots.txt as SitequiryBot.
• Readiness signals, not visibility. We did not measure whether a site is cited or shown in AI answers, and the presence of llms.txt or markup is not proof of any effect.
• One crawl from one location on 29 September 2026. Sites change, and we read up to 20 pages per site.
• Small groups. Language groups are small, and a single site moves a share by several points.
• No causal claims. The data shows what sites do, not why.

What this study does not measure

Each item below is left out for the reason given.

• Speed. Google PageSpeed was not run for this sample, so there are no Core Web Vitals, no distribution of mobile scores and no render-blocking or page-weight figures.
• Server response time. It was recorded for single requests from one location only; it is not a speed measurement.
• HTTPS redirects and security headers. The crawl did not record them.
• Visibility in AI answers. Only readiness signals are measured.
• Validity of structured data. It is counted by presence.
• Quality of llms.txt files. They are counted by presence.
• Sitequiry scores. The SEO, AEO, GEO and overall scores are our own weighted rubric, not an independent measure.
• Results per site. Only aggregates are published.
• Segments by site type or country. They would have fewer than 15 sites each.

Cite this study and download the data

You may quote the figures and reuse the charts with attribution. The aggregate data is licensed under Creative Commons Attribution (CC BY 4.0): you may share and adapt it, including commercially, if you credit GE-KO / Sitequiry and link to this page.

Summary for press and newsletters

What to do next

• Decide deliberately which AI crawlers you allow, and check your firewall and CDN too: AI crawlers and robots.txt.
• Add an llms.txt only if it is cheap to keep up to date: llms.txt: what it is and whether you need one.
• Describe who you are and what a page is about with accurate markup: structured data guide.
• Check your own site against these signals: free analysis, one URL, full result, no account.
• Want the fixes done? GE-KO implements them.

Frequently asked questions