The AI Crawler Block Index

Google’s AI Mode cites publishers that explicitly block AI crawlers in their robots.txt. In 10 queries we ran against Google AI Mode, 52 cited domains were in our sample of 10,894 top-web and publisher sites. 27 of those 52 (51.9%) block at least one AI crawler in robots.txt. In the sample as a whole, only 15% do. Blocking GPTBot doesn’t keep a site out of Google’s AI Overview.

That’s one of ten findings from the baseline, measured in July 2026 across 10,894 registrable domains (9,746 Tranco top-web + 1,148 news publishers). Nine others complicate the same picture. Cloudflare’s September 15, 2026 deadline covers 8.5% of the top web. Publishers block AI five times more than the open web, but 39.5% of GPTBot bans get ignored by the sites that set them. CDNs use different strategies. Fastly hard-blocks, Cloudflare challenges, CloudFront waves through. On publisher sites, the same datacenter IP served a browser 83.8% of the time and GPTBot 54.2%. llms.txt sits at 7.4% adoption. Cloudflare’s September 15 switch has since removed its managed AI notice from the sites that carried it, and the September re-run measures what moved.

What we measured

Two segments, one common sample. Tranco top 10,000 (list 46XLX, 2026-07-09) plus Ben Welsh’s news-homepages set (1,148 sites), deduplicated to 10,894 registrable domains. Scope and policy questions ran against all of them. That covers Cloudflare proxy presence, ad-tech signals, robots.txt entries for the AI user-agents, and llms.txt publication.

Enforcement questions ran against a 2,096-domain subset (the top 1,000 open web plus every publisher). Each domain got two requests from a US datacenter IP pinned to the same exit. One as the official GPTBot user-agent, one as a standard Chrome browser. A 600-domain sub-sample also got a residential IP pass so we could measure the datacenter-vs-residential gap.

The numbers at a glance:

Scope/policy rows cover all 10,894 domains. Enforcement rows come from the 2,096-domain subset.

The measurements use these definitions.

• Behind Cloudflare. The site’s response headers include cf-ray or server: cloudflare.
• Blocked (robots.txt). The bot’s own user-agent group contains Disallow: / with no overriding Allow. A global User-agent: * disallow does not count as AI-specific blocking.
• Paper-only ban. Robots.txt disallows GPTBot, but a live GPTBot request got a 200.
• Silent enforcer. Robots.txt does not disallow GPTBot, but GPTBot got blocked while the browser control got a 200 from the same IP.
• IP gap. GPTBot got blocked from a datacenter IP but served from a residential IP.

How the September re-run was measured

Same 10,894 domains, the same 2,096-site live test and the same 600-site residential sample, measured on September 16 between 11:50 and 13:46 UTC, after September 15 had ended in every US time zone. The definitions above are unchanged.

For the robots.txt decline we pulled archived copies from Common Crawl’s July and August crawls, covering the 165 Cloudflare sites that stopped disallowing GPTBot plus a control group of 150 that didn’t, and re-checked named examples against the live files. Common Crawl held copies for 133 of the 165.

Three limits are worth stating. This is change over time rather than a controlled experiment, and the one cause we name, Cloudflare replacing its managed feature, comes from Cloudflare’s own announcement. Our GPTBot requests carry the official user-agent string but are not OpenAI’s verified crawler, so a site matching on verified identity would treat them differently. The Google AI Mode test and the five-crawler residential test were not repeated.

Roughly 15% of enforcement requests hit proxy-level errors and were excluded from all rates. Another 8.2% hit a challenge or CAPTCHA and are reported as their own finding, not treated as noise. Blocking is easy to over-count if you conflate “our proxy pool got throttled” with “the site said no to GPTBot.” The browser-vs-GPTBot control from the same exit IP isolates identity as the variable.

Publishers vs the open web

Publishers block AI crawlers roughly five times more than the open web, and the gap holds for each bot individually. Whole-web GPTBot block rate is 7.9%. Publisher GPTBot block rate is 50.5%. ClaudeBot, CCBot, and Google-Extended show the same pattern at similar magnitudes, and the combined “blocks any AI crawler” rate is 10.3% vs 56.4%.

AI-crawler blocking is largely a publisher issue. Publishers have text that language models want to train on and cite from. They also have subscription revenue that competes directly with AI-generated summaries. Most e-commerce sites, blogs, and docs don’t perceive AI training as a direct business risk.

Any headline about “the web is closing to AI” really means “news publishers are closing to AI.” Applied to the whole top web, the number drops from 56.4% to 10.3%.

Two in five AI bans are paper-only

Of 592 sites in the enforcement subset that disallow GPTBot in robots.txt, 234 (39.5%) still served GPTBot a live 200 when it knocked. Open-web sites ignore their own robots.txt more often (46.7%) than publishers (38.9%). robots.txt is a suggestion, and 39.5% of the sites that added the block don’t enforce it themselves.

When a site does enforce something against GPTBot, the response varies.

• 22.2% return a hard block (403, 401, or 451)
• 10.9% return HTTP 402 Payment Required (Cloudflare’s pay-per-crawl signal)
• 5.0% serve a Cloudflare-style challenge or JavaScript interstitial
• 2.4% return HTTP 429 Rate Limited
• 2.3% present a CAPTCHA

The 402 signal is the newest development. Cloudflare’s pay-per-crawl program returns 402 when GPTBot lacks a paid arrangement. One in ten “blocks” isn’t really a block. It’s a paywall for bots expressed as an HTTP status.

The reverse case is the silent enforcer. 115 sites in the subset blocked GPTBot even though their robots.txt did not disallow it. Their firewall or WAF filters GPTBot by user-agent, but their robots.txt (where sites declare their AI policy) is silent about it. Declared policy and behavior diverge in both directions.

The GPTBot penalty

From a single US datacenter IP, publisher sites served a standard Chrome browser 83.8% of the time. The same IP, same domains, this time announcing the user-agent as GPTBot, got in only 54.2% of the time. A 30-point penalty for saying who you are. Whole-web sites show the same pattern with less spread (72.3% browser, 67.8% GPTBot).

Real enforcement is identity-based, not policy-based. Sites aren’t blocking IPs from OpenAI. They’re matching on the string in the User-Agent header. Any crawler that omits or lies about its identity walks in.

To check whether the penalty is specific to GPTBot or general to AI crawlers, we ran an extended test in July that the September re-run did not repeat. Five bots against 250 publisher domains from a residential vantage (a single machine with one consistent exit IP). Baseline browser access is lower on residential (77.2%) than on datacenter (83.8%), so the per-bot drops here are measured against the residential baseline.

ClaudeBot gets the hardest treatment at 36.8% pass, matched by CCBot. GPTBot and PerplexityBot get slightly better access. Meta-ExternalAgent sits between them.

Publishers signed content deals with OpenAI before Anthropic, which likely softened treatment for GPTBot. Perplexity frames itself as a search assistant, not a training crawler, and gets a marginal benefit of the doubt. Common Crawl’s downstream use case (indexing at web scale for research and indirectly for training) reads as unambiguous scraping to any WAF ruleset built in the last year. ClaudeBot has neither the political cover of a publisher deal nor the user-facing brand of a search assistant. It is a training bot with a company name.

The between-bot spread is smaller than the drop from browser. Any bot that names itself takes a 33 to 40 point hit from publishers. Change the User-Agent to a real browser string and access flips to the browser rate.

Different CDNs, different strategies

Fastly serves GPTBot 27.3% of the time and hard-blocks 61.5%. Cloudflare serves 51.3%, hard-blocks 16.9%, and adds a 24.7% JavaScript challenge layer. CloudFront serves 78.5% without intervention. The Cloudflare deadline dominates headlines, but Cloudflare isn’t the most aggressive gate.

Numbers are from the 2,096-domain enforcement subset.

Fastly returns a 403 the moment GPTBot connects. Cloudflare serves a Turnstile JavaScript challenge that only a browser client can execute. Akamai splits between hard blocks and rate limits. CloudFront passes 78.5% of GPTBot traffic without intervention.

The CDN in front of a target determines what “blocked” looks like on the wire. A scraper that handles 403 but not a JS challenge will see pass rates on the same site list vary by tens of percentage points depending on which vendor’s rules apply.

Fastly already hard-blocks at more than twice Cloudflare’s rate. No September 15 deadline announcement was needed.

Cloudflare’s new AI traffic defaults

On September 16 the default’s reach was 8.6% of the top web and 13.3% of publishers, within a rounding step of July.

The default block fires on Cloudflare-proxied sites that also carry ad-tech signals. Both conditions have to hold.

Only 26.0% of the top 10,000 sits behind Cloudflare’s proxy. Publisher share is 22.4%. Adding the ad-tech condition brings the addressable set down to 8.5% of the top web and 13.6% of publishers.

91.5% of the top web sits outside the deadline. Fastly, Amazon CloudFront, Akamai, and origin-only setups are unaffected by Cloudflare’s default.

The scope of the change still matters, because publishers dominate what AI Overview and AI Mode cite. A single publisher domain carries more training and grounding value than the average site in the top 10,000, so the addressable footprint understates the effect on what AI answers can actually source from.

The IP gap

Rotating IPs is the first thing a stalled scraper tries. On this dataset the gap is small. Of the 450 domains that blocked GPTBot from the datacenter IP, 48 served it from the residential one, a 10.7% escape rate. The September re-run puts that at 6.8%. One domain went the other way (served from datacenter, blocked from residential), likely a stale rate-limit rule.

The block layer matches on the User-Agent string. Changing exits does very little if the crawler still announces itself as GPTBot.

Policy and enforcement drift apart in both directions. Paper-only bans go one way, with robots.txt disallowing GPTBot and the origin serving it anyway. Silent enforcers go the other way. 115 sites (5.5% of the tested subset) blocked GPTBot even though their robots.txt did not disallow it. These sites publish permissive policy files and filter GPTBot at the CDN or WAF layer. The declared position and the wire behavior come from different systems and stay out of sync.

Challenges are a middle state between pass and block on the wire. 8.2% of enforcement measurements returned a Cloudflare Turnstile or JavaScript interstitial. A bare HTTP client sees a 200 OK on a page it cannot parse, or a “please wait” body that never resolves.

Public “AI crawler blocking” counts often fold these categories into either the pass or the block bucket, which is why the same underlying data produces very different headline numbers depending on how each layer gets classified.

Blocked domains still show up in Google’s AI Mode

Measured in July and not repeated in September.

Ten queries against Google AI Mode across a general news mix returned 82 unique cited domains. 52 of those overlap with the sample. 27 of the 52 (51.9%) disallow at least one AI crawler in robots.txt. Across the sample as a whole, the same rate is 15%. Sites that block AI crawlers get cited by Google AI Mode more than three times as often as the sample baseline.

Ten major blockers by citation count:

The mechanism is boring. Google-Extended is a separate directive from GPTBot. A site can disallow OpenAI’s crawler, Anthropic’s crawler, Common Crawl, and Perplexity, and Google’s grounding surfaces will still pull from Google’s own index, which the site isn’t blocking (blocking it would drop the domain from regular search too).

Sites that also disallow Google-Extended can still appear in AI Overview through Google’s normal search cache, which pre-dates the Google-Extended directive.

For an SEO or content team, blocking GPTBot is a policy statement about training and third-party crawlers. Appearance in Google’s AI answers is a separate toggle, controlled by Google-Extended and by what already sits in Google’s search cache.

llms.txt adoption, 7.4% in July and 8.8% in September

llms.txt is a proposed spec for publishing an LLM-oriented site map, a curated list of URLs and short descriptions that language models can pull in place of scraping the full site. Adoption is 7.4% across the whole 10,894-domain sample.

The split is inverted from robots.txt blocking:

Publishers block AI crawlers at 5.5x the whole-web rate and publish llms.txt at 40% of it. Blocking is a policy statement that costs nothing to write. llms.txt is curation work aimed at making a site more useful to LLMs, which is the outcome publishers are trying to prevent.

llms.txt also has no enforcement side. OpenAI, Anthropic, Google, and Perplexity all read robots.txt. None of them have committed to reading llms.txt or preferring its listing over their own crawl of the site. Publishing it is an advisory signal a crawler is free to ignore.

For a docs site or a content property that wants LLMs to answer questions accurately from its content, llms.txt is one of the cheapest signals available. For a publisher trying to control how its content is used in AI answers, robots.txt and CDN-level rules already cover the enforcement side.

The September re-run

Cloudflare turned on its new AI bot rules on September 15. We re-measured the same 10,894 websites the next day. Blocking held roughly where it was. The share of sites announcing it in robots.txt fell from 15.2% to 13.0%.

Cloudflare’s managed robots.txt block, the one that wrote “ai-train=no” and a GPTBot ban into customers’ files on their behalf, is gone from the sites that carried it. We found it on 118 Cloudflare sites in the archived copies Common Crawl held for them. On September 16 it was on none of those sites. Its announced replacement, Bot Preference Sync, appeared in none of the 6,451 robots.txt files we collected, because existing customers have to review and confirm it before it writes anything.

Of the 21 affected sites that also ran in our live test, 16 still blocked or challenged GPTBot. What went missing is the machine-readable “no”, the part a crawler reads before it decides whether to fetch anything.

What the September 15 switch removed

Among sites behind Cloudflare, the share disallowing GPTBot in robots.txt fell from 17.1% to 9.9%. The same measurement on sites not behind Cloudflare moved by a tenth of a point, 18.7% to 18.6%. Both groups are the same sites in both runs, so this is change within each group rather than a different sample.

To see what the files looked like before, we pulled archived copies from Common Crawl for the sites that stopped disallowing GPTBot. 118 of them carried Cloudflare’s managed block in their archived copy, 107 taken from the August crawl and 11 from July. All 118 said the same three things, a Content-Signal line with ai-train=no, a rights reservation citing Article 4 of the EU copyright directive, and a GPTBot ban. Among them are the Smithsonian, Ko-fi, AirAsia, Find a Grave, Vatican News, National Review, Punchbowl News, Bellingcat, Charlie Hebdo and Novaya Gazeta.

A live re-check of all 118 the following day found an AI-training signal on one of the 101 that answered. Patreon carries a Content-Signal line with ai-train=no, scoped to two named crawlers and written by Patreon rather than by Cloudflare. A site that wrote its own rules still has them, and a site that let Cloudflare write them has an empty file where they were.

Cloudflare describes the change as replacing Managed Robots.txt with Bot Preference Sync, and its engineering post says legacy users are prompted to review and confirm the new feature before it takes effect. Between the removal and that confirmation, the file says nothing.

July 10 against September 16

Every row below is the same panel measured twice, so a change in it is a change on those sites rather than a different set of sites.

*The open-web paper-only share rests on 44 to 45 sites, which is too small a base to carry a headline.

Publishers moved in both directions at once. Fewer of them wrote a GPTBot ban, and a larger share of the bans that remain are enforced.

The browser row is the control, and it barely moved. So the distance between what a browser gets and what GPTBot gets on the same sites widened from 29.6 points in July to 35.9 in September, which is a change in how those sites treat the crawler rather than in how reachable they are.

The new blocking is mostly not Cloudflare’s

In the live test of 2,096 sites, 125 started blocking GPTBot between the two runs and 66 stopped. 112 of the 125 are news publishers. Only 22 of them sit behind Cloudflare, so the default that switched on in September accounts for a small part of the movement.

Sites that block GPTBot without saying so in robots.txt grew from 115 to 178. That group is the one a crawler can’t anticipate, because the file it reads gives no warning of the block it’s about to meet.

What to check if your site is behind Cloudflare

Open your own /robots.txt and read it. If your AI crawler rules came from Cloudflare’s managed feature, they are probably not there any more.

Bot Preference Sync writes to robots.txt as well, and for an existing customer it waits for a review and a confirmation in the Cloudflare dashboard. Until someone does that, the file carries whatever is left after the old block was removed.

Check that robots.txt itself is reachable. 125 Cloudflare sites in our sample served it to us in July and refused it in September, and a file behind a bot challenge is a file no crawler can obey.

Search and AI training are separate decisions, and Cloudflare now exposes them as separate settings. A site that wants to stay in search results while refusing training data has to say so in both places.

What the numbers add up to

Robots.txt carries the declared policy. The CDN or WAF layer does the actual enforcement. The User-Agent header is what that enforcement matches on. They often disagree, and the data quantifies where.

39.5% of sites that disallow GPTBot in robots.txt served it a live 200 from a datacenter IP. 5.5% of sites that stay silent about GPTBot block it at the CDN layer anyway. About 8% of enforcement responses are challenges. Turnstile pages or JavaScript interstitials get served in place of a clean status code.

The observable outcome for a given crawler on a given site depends on which CDN is in front, which User-Agent the client sends, which page was requested, and whether the site actually enforces what robots.txt says. Any single-number headline about “how much of the web blocks GPTBot” collapses these variables into one axis.

The paired-request pattern used in this baseline is the only clean way to isolate identity as the variable. A single request with a bot User-Agent, without a browser control from the same IP, cannot distinguish “this site blocks GPTBot” from “our proxy pool is being rate-limited by this CDN.”

IP class is a weak lever. 10.7% of datacenter-blocked domains served GPTBot from a residential IP in July, and 6.8% in September. Changing the User-Agent to a real browser recovers a much larger share.

FAQ

Does blocking GPTBot in robots.txt keep my site out of ChatGPT? Partly. GPTBot is OpenAI’s dedicated crawler for training data and browsing plugins. Blocking it stops those specific pulls. It doesn’t stop OpenAI’s models from citing content that reached them through Common Crawl, Bing’s index (used by SearchGPT), or licensed data deals.

What’s the difference between GPTBot and Google-Extended? GPTBot is OpenAI’s directive controlling whether OpenAI’s crawler can fetch your pages. Google-Extended is Google’s directive controlling whether pages already in Google’s search index can be used to ground AI Overview and AI Mode answers. Blocking one has no effect on the other.

Is HTTP 402 Payment Required a real block? It is Cloudflare’s pay-per-crawl signal. Cloudflare returns 402 to bots without a paid arrangement. 10.9% of GPTBot enforcement responses in the sample were 402s. For a bot without a deal, the outcome is the same as a hard block.

Does llms.txt prevent AI training? No. llms.txt is an opt-in advisory format for publishing an LLM-oriented site map, a curated list of URLs and descriptions. No major AI crawler treats it as authoritative or restricts access based on it. To keep AI systems out, robots.txt and WAF rules are the enforcement tools.

What is Bot Preference Sync? Cloudflare’s replacement for its Managed Robots.txt feature. It writes AI preferences into robots.txt the way the old feature did, and for an existing customer it waits for a review and a confirmation in the dashboard before it writes anything. On September 16 it appeared in none of the 6,451 robots.txt files we collected.

How do I check whether my site lost Cloudflare’s AI notice? Open your own /robots.txt and look for a Content-Signal line with ai-train=no and a User-agent: GPTBot group. If your rules came from Cloudflare’s managed feature and neither is there, the block was removed and its replacement is waiting on a confirmation in your Cloudflare dashboard.

Should I use IP rotation to bypass AI crawler blocks? IP rotation recovered 10.7% of blocked domains in July and 6.8% in September. Enforcement matches on the User-Agent string much more consistently than on IP class. From a fixed IP, requests with a real browser User-Agent got served at rates 20 to 30 points higher than requests with the GPTBot User-Agent.

Conclusion

Declared policy and observed enforcement disagree on the AI-crawler question. About one in ten top-web sites explicitly blocks AI crawlers in robots.txt, and publishers do it at more than five times that rate. Two in five of those bans don’t enforce on the wire, while 5.5% of the sample enforces without declaring anything. CDN choice determines what “blocked” looks like as a response class. Google’s own AI answers keep citing sites that disallow AI crawlers, because the directives that gate them are on different toggles.

Cloudflare’s September 15 default block covers 8.5% of the top web. Publishers dominate what AI Overview and AI Mode cite from, which means the effective impact runs larger than the addressable footprint suggests.