The AI Crawler Block Index

Google’s AI Mode cites publishers that explicitly block AI crawlers in their robots.txt. In 10 queries we ran against Google AI Mode, 52 cited domains were in our sample of 10,880 top-web and publisher sites. 27 of those 52 (51.9%) block at least one AI crawler in robots.txt. In the sample as a whole, only 15% do. Blocking GPTBot does not keep a site out of Google’s AI Overview.

That’s one of ten findings from the baseline, measured in July 2026 across 10,894 registrable domains (9,746 Tranco top-web + 1,148 news publishers). Nine others complicate the same picture. Cloudflare’s September 15, 2026 deadline covers 8.5% of the top web. Publishers block AI five times more than the open web, but 39.5% of GPTBot bans get ignored by the sites that set them. CDNs use different strategies. Fastly hard-blocks, Cloudflare challenges, CloudFront waves through. Same datacenter IP, browser served 83.8%, GPTBot 54.2%. llms.txt sits at 7.4% adoption.

This is the July 2026 baseline, measured before Cloudflare’s September 15 default block took effect.

What we measured

Two segments, one common sample. Tranco top 10,000 (list 46XLX, 2026-07-09) plus Ben Welsh’s news-homepages set (1,148 sites), deduplicated to 10,894 registrable domains. Scope and policy questions ran against all of them. That covers Cloudflare proxy presence, ad-tech signals, robots.txt entries for the AI user-agents, and llms.txt publication.

Enforcement questions ran against a 2,096-domain subset (the top 1,000 open web plus every publisher). Each domain got two requests from a US datacenter IP pinned to the same exit. One as the official GPTBot user-agent, one as a standard Chrome browser. A 600-domain sub-sample also got a residential IP pass so we could measure the datacenter-vs-residential gap.

The numbers at a glance:

Scope/policy rows cover all 10,894 domains. Enforcement rows come from the 2,096-domain subset.

The measurements use these definitions.

• Behind Cloudflare. The site’s response headers include cf-ray or server: cloudflare.
• Blocked (robots.txt). The bot’s own user-agent group contains Disallow: / with no overriding Allow. A global User-agent: * disallow does not count as AI-specific blocking.
• Paper-only ban. Robots.txt disallows GPTBot, but a live GPTBot request got a 200.
• Silent enforcer. Robots.txt does not disallow GPTBot, but GPTBot got blocked while the browser control got a 200 from the same IP.
• IP gap. GPTBot got blocked from a datacenter IP but served from a residential IP.

Roughly 15% of enforcement requests hit proxy-level errors and were excluded from all rates. Another 8.2% hit a challenge or CAPTCHA and are reported as their own finding, not treated as noise. Blocking is easy to over-count if you conflate “our proxy pool got throttled” with “the site said no to GPTBot.” The browser-vs-GPTBot control from the same exit IP isolates identity as the variable.

Publishers vs the open web

Publishers block AI crawlers roughly five times more than the open web, and the gap holds for each bot individually. Whole-web GPTBot block rate is 7.9%. Publisher GPTBot block rate is 50.5%. ClaudeBot, CCBot, and Google-Extended show the same pattern at similar magnitudes, and the combined “blocks any AI crawler” rate is 10.3% vs 56.4%.

AI-crawler blocking is largely a publisher issue. Publishers have text that language models want to train on and cite from. They also have subscription revenue that competes directly with AI-generated summaries. Most e-commerce sites, blogs, and docs don’t perceive AI training as a direct business risk.

Any headline about “the web is closing to AI” really means “news publishers are closing to AI.” Applied to the whole top web, the number drops from 56.4% to 10.3%.

Two in five AI bans are paper-only

Of 592 sites in the enforcement subset that disallow GPTBot in robots.txt, 234 (39.5%) still served GPTBot a live 200 when it knocked. Open-web sites ignore their own robots.txt more often (46.7%) than publishers (38.9%). robots.txt is a suggestion, and 39.5% of the sites that added the block do not enforce it themselves.

When a site does enforce something against GPTBot, the response varies.

• 22.2% return a hard block (403, 401, or 451)
• 10.9% return HTTP 402 Payment Required (Cloudflare’s pay-per-crawl signal)
• 5.0% serve a Cloudflare-style challenge or JavaScript interstitial
• 2.4% return HTTP 429 Rate Limited
• 2.3% present a CAPTCHA

The 402 signal is the newest development. Cloudflare’s pay-per-crawl program returns 402 when GPTBot lacks a paid arrangement. One in ten “blocks” is not really a block. It is a paywall for bots expressed as an HTTP status.

The reverse case is the silent enforcer. 115 sites in the subset blocked GPTBot even though their robots.txt did not disallow it. Their firewall or WAF filters GPTBot by user-agent, but their robots.txt (where sites declare their AI policy) is silent about it. Declared policy and behavior diverge in both directions.

The GPTBot penalty

From a single US datacenter IP, publisher sites served a standard Chrome browser 83.8% of the time. The same IP, same domains, this time announcing the user-agent as GPTBot, got in only 54.2% of the time. A 30-point penalty for saying who you are. Whole-web sites show the same pattern with less spread (72.3% browser, 67.8% GPTBot).

Real enforcement is identity-based, not policy-based. Sites are not blocking IPs from OpenAI. They are matching on the string in the User-Agent header. Any crawler that omits or lies about its identity walks in.

To check whether the penalty is specific to GPTBot or general to AI crawlers, we ran an extended test. Five bots against 250 publisher domains from a residential vantage (a single machine with one consistent exit IP). Baseline browser access is lower on residential (77.2%) than on datacenter (83.8%), so the per-bot drops here are measured against the residential baseline.

ClaudeBot gets the hardest treatment at 36.8% pass, matched by CCBot. GPTBot and PerplexityBot get slightly better access. Meta-ExternalAgent sits between them.

Publishers signed content deals with OpenAI before Anthropic, which likely softened treatment for GPTBot. Perplexity frames itself as a search assistant, not a training crawler, and gets a marginal benefit of the doubt. Common Crawl’s downstream use case (indexing at web scale for research and indirectly for training) reads as unambiguous scraping to any WAF ruleset built in the last year. ClaudeBot has neither the political cover of a publisher deal nor the user-facing brand of a search assistant. It is a training bot with a company name.

The between-bot spread is smaller than the drop from browser. Any bot that names itself takes a 33 to 40 point hit from publishers. Change the User-Agent to a real browser string and access flips to the browser rate.

Different CDNs, different strategies

Fastly serves GPTBot 27.3% of the time and hard-blocks 61.5%. Cloudflare serves 51.3%, hard-blocks 16.9%, and adds a 24.7% JavaScript challenge layer. CloudFront serves 78.5% without intervention. The Cloudflare deadline dominates headlines, but Cloudflare is not the most aggressive gate.

Numbers are from the 2,096-domain enforcement subset.

Fastly returns a 403 the moment GPTBot connects. Cloudflare serves a Turnstile JavaScript challenge that only a browser client can execute. Akamai splits between hard blocks and rate limits. CloudFront passes 78.5% of GPTBot traffic without intervention.

The CDN in front of a target determines what “blocked” looks like on the wire. A scraper that handles 403 but not a JS challenge will see pass rates on the same site list vary by tens of percentage points depending on which vendor’s rules apply.

Fastly already hard-blocks at more than twice Cloudflare’s rate. No September 15 deadline announcement was needed.

Cloudflare’s new AI traffic defaults

The default block fires on Cloudflare-proxied sites that also carry ad-tech signals. Both conditions have to hold.

Only 26.0% of the top 10,000 sits behind Cloudflare’s proxy. Publisher share is 22.4%. Adding the ad-tech condition brings the addressable set down to 8.5% of the top web and 13.6% of publishers.

91.5% of the top web sits outside the deadline. Fastly, Amazon CloudFront, Akamai, and origin-only setups are unaffected by Cloudflare’s default.

The scope of the change still matters, because publishers dominate what AI Overview and AI Mode cite. A single publisher domain carries more training and grounding value than the average site in the top 10,000, so the addressable footprint understates the effect on what AI answers can actually source from.

The IP gap

Rotating IPs is the first thing a stalled scraper tries. On this dataset the gap is small. 310 domains blocked GPTBot from the datacenter IP. Only 11 of them served GPTBot from the residential IP, a 3.5% escape rate. One domain went the other way (served from datacenter, blocked from residential), likely a stale rate-limit rule.

The block layer matches on the User-Agent string. Changing exits does very little if the crawler still announces itself as GPTBot.

Policy and enforcement drift apart in both directions. Paper-only bans go one way, with robots.txt disallowing GPTBot and the origin serving it anyway. Silent enforcers go the other way. 115 sites (5.5% of the tested subset) blocked GPTBot even though their robots.txt did not disallow it. These sites publish permissive policy files and filter GPTBot at the CDN or WAF layer. The declared position and the wire behavior come from different systems and stay out of sync.

Challenges are a middle state between pass and block on the wire. 8.2% of enforcement measurements returned a Cloudflare Turnstile or JavaScript interstitial. A bare HTTP client sees a 200 OK on a page it cannot parse, or a “please wait” body that never resolves.

Public “AI crawler blocking” counts often fold these categories into either the pass or the block bucket, which is why the same underlying data produces very different headline numbers depending on how each layer gets classified.

Blocked domains still show up in Google’s AI Mode

Ten queries against Google AI Mode across a general news mix returned 82 unique cited domains. 52 of those overlap with the sample. 27 of the 52 (51.9%) disallow at least one AI crawler in robots.txt. Across the sample as a whole, the same rate is 15%. Sites that block AI crawlers get cited by Google AI Mode more than three times as often as the sample baseline.

Ten major blockers by citation count:

The mechanism is boring. Google-Extended is a separate directive from GPTBot. A site can disallow OpenAI’s crawler, Anthropic’s crawler, Common Crawl, and Perplexity, and Google’s grounding surfaces will still pull from Google’s own index, which the site is not blocking (blocking it would drop the domain from regular search too).

Sites that also disallow Google-Extended can still appear in AI Overview through Google’s normal search cache, which pre-dates the Google-Extended directive.

For an SEO or content team, blocking GPTBot is a policy statement about training and third-party crawlers. Appearance in Google’s AI answers is a separate toggle, controlled by Google-Extended and by what already sits in Google’s search cache.

llms.txt adoption is stuck at 7.4%

llms.txt is a proposed spec for publishing an LLM-oriented site map, a curated list of URLs and short descriptions that language models can pull in place of scraping the full site. The idea gets plenty of coverage. “llms.txt” pulls around 15,000 US monthly searches. Adoption sits at 7.4% across the whole 10,894-domain sample.

The split is inverted from robots.txt blocking:

Publishers block AI crawlers at 5.5x the whole-web rate and publish llms.txt at 40% of it. Blocking is a policy statement that costs nothing to write. llms.txt is curation work aimed at making a site more useful to LLMs, which is the outcome publishers are trying to prevent.

llms.txt also has no enforcement side. OpenAI, Anthropic, Google, and Perplexity all read robots.txt. None of them have committed to reading llms.txt or preferring its listing over their own crawl of the site. Publishing it is an advisory signal a crawler is free to ignore.

For a docs site or a content property that wants LLMs to answer questions accurately from its content, llms.txt is one of the cheapest signals available. For a publisher trying to control how its content is used in AI answers, robots.txt and CDN-level rules already cover the enforcement side.

What the numbers add up to

Robots.txt carries the declared policy. The CDN or WAF layer does the actual enforcement. The User-Agent header is what that enforcement matches on. They often disagree, and the data quantifies where.

39.5% of sites that disallow GPTBot in robots.txt served it a live 200 from a datacenter IP. 5.5% of sites that stay silent about GPTBot block it at the CDN layer anyway. About 8% of enforcement responses are challenges. Turnstile pages or JavaScript interstitials get served in place of a clean status code.

The observable outcome for a given crawler on a given site depends on which CDN is in front, which User-Agent the client sends, which page was requested, and whether the site actually enforces what robots.txt says. Any single-number headline about “how much of the web blocks GPTBot” collapses these variables into one axis.

The paired-request pattern used in this baseline is the only clean way to isolate identity as the variable. A single request with a bot User-Agent, without a browser control from the same IP, cannot distinguish “this site blocks GPTBot” from “our proxy pool is being rate-limited by this CDN.”

IP class is a weak lever. 3.5% of datacenter-blocked domains served GPTBot from a residential IP. Changing the User-Agent to a real browser recovers a much larger share.

FAQ

Does blocking GPTBot in robots.txt keep my site out of ChatGPT? Partly. GPTBot is OpenAI’s dedicated crawler for training data and browsing plugins. Blocking it stops those specific pulls. It does not stop OpenAI’s models from citing content that reached them through Common Crawl, Bing’s index (used by SearchGPT), or licensed data deals.

What’s the difference between GPTBot and Google-Extended? GPTBot is OpenAI’s directive controlling whether OpenAI’s crawler can fetch your pages. Google-Extended is Google’s directive controlling whether pages already in Google’s search index can be used to ground AI Overview and AI Mode answers. Blocking one has no effect on the other.

Is HTTP 402 Payment Required a real block? It is Cloudflare’s pay-per-crawl signal. Cloudflare returns 402 to bots without a paid arrangement. 10.9% of GPTBot enforcement responses in the sample were 402s. For a bot without a deal, the outcome is the same as a hard block.

Does llms.txt prevent AI training? No. llms.txt is an opt-in advisory format for publishing an LLM-oriented site map, a curated list of URLs and descriptions. No major AI crawler treats it as authoritative or restricts access based on it. To keep AI systems out, robots.txt and WAF rules are the enforcement tools.

Should I use IP rotation to bypass AI crawler blocks? IP rotation recovers only 3.5% of blocked domains in the sample. Enforcement matches on the User-Agent string much more consistently than on IP class. From a fixed IP, requests with a real browser User-Agent got served at rates 20 to 30 points higher than requests with the GPTBot User-Agent.

Conclusion

Declared policy and observed enforcement do not line up on the AI-crawler question. About one in ten top-web sites explicitly blocks AI crawlers in robots.txt, and publishers do it at more than five times that rate. Two in five of those bans do not enforce on the wire, while 5.5% of the sample enforces without declaring anything. CDN choice determines what “blocked” looks like as a response class. Google’s own AI answers keep citing sites that disallow AI crawlers, because the directives that gate them are on different toggles.

Cloudflare’s September 15 default block covers 8.5% of the top web. Publishers dominate what AI Overview and AI Mode cite from, which means the effective impact runs larger than the addressable footprint suggests.