The AI Hypocrisy Index

The AI Hypocrisy Index measures the gap between what a company writes in its Terms of Service and what it does with data. We scored 35 AI companies and the 150 most-visited consumer websites on two 0-to-10 scales. The forbid score measures how hard the Terms ban scraping. The take score measures how broadly the company claims user or public data, and their product divided by ten is the Index. It only climbs when a company scores high on both.

Adobe, Perplexity, and LinkedIn top the Index. All three ban scraping in strong Terms language, yet their robots.txt blocks none of the tracked AI training crawlers. Five of the top ten companies follow that pattern.

19 of 35 AI companies write an explicit anti-scraping ban into their Terms while claiming rights to user or public data. 85 of the 150 top consumer sites carry at least one anti-scraping clause, and 61 of them an explicit ban. 47 times a platform’s robots.txt disallows a rival AI company’s crawler.

X Corp’s Terms set liquidated damages of €15,000 per million posts once a scraper passes a million posts a day. Nine companies train on user content by default, requiring an explicit opt-out. Adobe scores 4.4 out of 10 and stays in the top five under every alternative weighting scheme we tried. 17 firms share nearly identical “robot, spider, scraper” phrasing across their Terms of Service.

This is the July 2026 snapshot. Every claim is backed by an exact-substring-verified quote from the source document.

What we measured

We split the sample into two tiers. Tier A is 35 companies that build AI models or operate crawlers. For each we read the Terms of Service, Privacy Policy, and robots.txt. Tier B is the 150 most-visited consumer websites drawn from the Tranco top ranking (list 94Y82). For Tier B we read Terms of Service and robots.txt only. HasData and its own domain were excluded from the sample.

Documents were fetched on 23 July 2026 with a custom User-Agent (DoubleStandardIndexResearch/1.0), at one request per second per domain. No paywalls, logins, or bot protections were circumvented.

Each document was classified against a 12-clause taxonomy, split into 7 anti-scraping clauses and 5 extraction clauses. An LLM classified each document at temperature zero, returning strict JSON. Every reported clause carries a verbatim quotation programmatically verified as an exact substring of the source document. Classification was run twice. 402 findings passed at a 100% exact-substring rate. 12 findings failed verification and were dropped.

Each Tier-A company gets two scores on a 0 to 10 scale. The forbid score counts anti-scraping clauses. The take score counts extraction clauses. The AI Hypocrisy Index is their product divided by ten. A pure blocker or a pure scraper stays low. The Index climbs only when a company scores high on both sides.

The numbers at a glance:

Privacy Policies weren’t read for Tier B, so the following apply to Tier A only. 9 companies train on user content by default (opt-out). 13 companies describe collecting publicly available web data in their Privacy Policy. 1 company has a liquidated-damages clause (X Corp).

The measurements use these definitions.

• Explicit scraping ban. Terms of Service contain the general scraping ban (clause AS1).
• Anti-scraping clause. Terms of Service contain at least one of the 7 AS clauses (general scraping ban, absolute ban, prior written consent, AI-training ban on their content, liquidated damages, anti-circumvention, API-only access).
• AI training crawler. A user-agent operated by an AI vendor for training-data collection. 11 were tracked, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, and Meta-ExternalAgent.
• Paper-only forbidder. Terms declare an anti-scraping stance (forbid score ≥ 2.5) and robots.txt blocks zero of the 11 tracked AI training crawlers.
• Silent enforcer. Terms are permissive (forbid score < 2.5) and robots.txt blocks at least one AI training crawler.
• Verbatim verification. A reported clause is included only if its exact wording is found as an exact substring of the fetched source document.

One of 35 Tier-A Terms pages was a JavaScript-only site and couldn’t be read in text form. The historical (Wayback) dimension is partial and is used for context only.

The AI Hypocrisy Index leaderboard

Adobe, Perplexity, and LinkedIn top the list. Every one of them tells you not to scrape their content and blocks zero AI training crawlers in robots.txt.

Adobe processes your documents and images through AI, claims rights over what you upload, and tells everyone else that scraping Adobe isn’t allowed. Forbid 5.5, take 8.

Perplexity scored 6.5 on the forbid component, the highest of any company in the sample that built its product by indexing the web. Its robots.txt blocks zero AI training crawlers.

Table 1 lists the top 15 Tier-A companies with their forbid score, take score, composite Index, and the number of AI training crawlers each blocks in its own robots.txt.

Anthropic trains on your Claude conversations by default. Take 5, forbid 0, composite 0.0. GitHub claims rights over code in public repositories. Take 9, the highest in the sample, forbid 0, composite 0.0.

The forbid-vs-take split

The content platforms own the upper-right of this chart. Adobe, Perplexity, Reddit, and Yelp all restrict access aggressively in their Terms and claim broad rights over the content on their platforms.

OpenAI and Google both score 2.5 on forbid and 7 on take, describe collecting “publicly available” data, and run training crawlers.

X Corp and LinkedIn score high on forbid (5.0 and 6.5) but only 4 on take. They enforce access restrictions but claim less from user content than Reddit or Yelp.

Anthropic, Microsoft, Apple, Nvidia, Salesforce/Slack, xAI, Mistral, Stability AI, and Stack Overflow all score 0 on forbid. They collect data and place no legal restrictions on scraping. Composite 0.0.

Banning scraping is the norm

Beyond the 19 AI companies with an explicit ban, four more carry a narrower anti-scraping clause, which makes 23 of the 35. The consumer tier leans further, with 85 of the 150 sites carrying at least one clause. A written scraping ban is the default across the top sites in the sample.

The AI builders are in this group too. Google’s Terms prohibit “scraping content that doesn’t belong to you.” OpenAI bans “automatically or programmatically extracting data or Output.” Both companies also describe collecting “information that is publicly available on the internet” to train their models.

Seven companies banned scraping in their Terms and described collecting public web data in their Privacy Policy. Adobe, Booking, Getty Images, Google, LinkedIn, OpenAI, and Yelp. Both documents are on the same domain.

AI labs block each other in robots.txt

Reading the robots.txt of 24 AI companies against every rival’s crawler user-agents, we found 47 pairs where a platform disallows at least one crawler of a rival AI company. Reddit, Yelp, and Meta each block all 11 tracked AI training bots in their robots.txt, the most in Tier A. X Corp, Pinterest, and Quora each block 10. Amazon blocks 9, and ByteDance/TikTok blocks 8.

Which platform blocks which rival is visible in the matrix. X Corp’s robots.txt disallows OpenAI, Anthropic, Perplexity, and seven others. Meta disallows ten rivals. All 47 pairs come from five of the 24 companies in the matrix (X Corp, Meta, Amazon, Quora, and ByteDance/TikTok), and the other 19 don’t disallow any rival crawler site-wide.

Every AI vendor we tracked appears on someone’s disallow list. OpenAI, Anthropic, Google, Perplexity, Apple, Amazon, Meta, ByteDance, Mistral, Cohere, Common Crawl, and Microsoft all get blocked by at least one platform. The phrase “publicly available” describes data that’s available to everyone except the specific crawlers each platform wants to keep out.

Where robots.txt disagrees with Terms

The Terms of Service and the robots.txt are separate signals from the same company. In this sample, they disagree in both directions.

37% of the sample writes anti-scraping into their Terms while leaving the machine-readable signal open. At these 13 companies, the declared position and the wire-level policy don’t match.

Perplexity, LinkedIn, and Adobe are the strongest examples. Perplexity’s Terms score 6.5 on the forbid component. LinkedIn scores 6.5. Adobe scores 5.5. All three block zero AI training crawlers in their own robots.txt.

Pinterest and Quora reverse this. Both have permissive Terms (forbid 0) but their robots.txt blocks 10 of the 11 tracked AI training crawlers.

Companies that scrape after suing over scraping

LinkedIn spent years in court against hiQ Labs, a startup that scraped public LinkedIn profiles. The case reached the 9th Circuit in 2022. LinkedIn’s User Agreement forbids anyone else from using “crawlers, browser plugins and add-ons or any other technology” to access the platform. LinkedIn’s own Privacy Policy states that the company “may collect public information about you, such as professional-related news and accomplishments, and make it available as part of our Services.”

Getty Images filed suit against Stability AI in the UK High Court over training an image model on Getty’s library. The court ruled in 2025. Getty’s own Terms forbid “using any data mining, robots or similar data gathering or extraction methods.” The same behavior Getty accused Stability AI of is banned in Getty’s Terms for anyone else.

Reddit licensed its user content to Google for a reported $60 million a year, then sued Perplexity for accessing the same data without a contract. Reddit’s Terms grant Reddit “the right to use Your Content to train AI and machine learning models.”

X Corp’s liquidated damages clause

X Corp is the only company in the sample with a liquidated-damages clause. Anyone “requesting, viewing, or accessing more than 1,000,000 posts” in any 24-hour period owes €15,000 per 1,000,000 posts under the version of the Terms we read for the study. The archived 2025 copy of the same page names $15,000.

X’s Terms authorize the company to sue any party that “access or search or attempt to access or search the Services by any means (automated or otherwise) other than through our currently available, published interfaces.” At the same time, X’s Privacy Policy says the company “may use the information we collect and publicly available information to help train our machine learning or artificial intelligence models.”

34 of 35 Tier-A companies leave enforcement to general remedies (injunction, actual damages), and only X Corp names a fixed amount.

Nine companies train on user content by default

Nine of the 35 Tier-A companies train AI models on their users’ content on an opt-out basis. Adobe, Anthropic, DeepSeek, GitHub, Microsoft, Mistral, OpenAI, Quora, and Stability AI all keep the behavior enabled by default. A user has to change the setting explicitly.

Anthropic’s Privacy Policy reads, “We may use your Inputs and Outputs to train and improve Anthropic AI models, unless you opt out through your account settings.” Microsoft’s Terms carry a parallel clause. “This data can help train our AI models in Microsoft Copilot unless you opt out.”

The scope varies by product. Adobe’s clause covers documents and images processed through its services. GitHub’s covers code and issues in public repositories (with different rules for private ones). Microsoft’s covers Copilot interactions and Microsoft 365 data. OpenAI’s covers ChatGPT conversations. Anthropic’s covers Claude conversations.

Clause language patterns across the industry

The 402 verified anti-scraping quotes cluster into a small number of legal templates. Companies are copying each other’s language.

The “robot, spider, scraper” template covers 17 sites. Netflix’s Terms include “use any robot, spider, scraper or other automated means to access the Netflix service.” Perplexity’s use “use any robot, spider, crawlers, scraper, or other automatic device, process, software or queries.” Expedia, Shopify, Snap, Airbnb, Discord, Unsplash, and Scribd all carry near-identical phrasings.

The “data mining, robots or similar data gathering” template covers 13 sites. Adobe’s clause reads “use any data mining or similar data gathering and extraction methods.” Cohere’s reads “use any data mining, robots, or similar data gathering or extraction methods.” Getty Images, IMDB, Forbes, Britannica, Duolingo, Goodreads, and Walmart share the same construction.

The AI-training-ban template covers 11 sites. Amazon’s Terms carry “You will not, and will not allow any third party to, use AI-generated content from the Amazon Services to, directly or indirectly, develop or improve large language or multimodal models.” Intuit’s Terms include “scraping, accessing, or downloading content that doesn’t belong to you, or training or developing artificial intelligence or machine learning models.” The Guardian, Time, Financial Times, and Unsplash use variations of the same clause.

The anti-circumvention template covers 12 sites. Netflix’s clause reads “circumvent, remove, alter, deactivate, degrade, block, obscure or thwart any of the content protections or other elements of the Netflix service.” Perplexity’s reads “circumvent, remove, alter, deactivate, degrade or thwart any technological measure or content protections of the Services.” Common Crawl, Getty, Time, Scribd, Slideshare, and Cisco carry near-identical language.

Terms of Service in this sample rely on shared legal templates. Individual companies vary the exact wording but the underlying construction is the same across each cluster. Anti-scraping language is adopted by default from industry norms, while extraction clauses are drafted separately for each company’s business model.

The rankings hold under alternative weights

The composite Index is the product of the forbid and take scores divided by ten. Because every scoring component is preserved as its own column, the same data can be re-weighted. Four alternative weightings produce the following top-5 lists.

Heatmap:

Under the equal-weight sum, Yelp moves up two positions and GitHub jumps from 34th to 10th because its high take score (9) is no longer multiplied by a zero forbid. Dropping AS5 barely changes the ordering, since only X Corp has an AS5 clause.

Forbid-only isolates the sites with the strictest Terms. Perplexity leads at 6.5, LinkedIn at 6.5, Adobe at 5.5. Midjourney, Expedia, and Airbnb all move into the top-7 at forbid 4.5 each. Reddit, Amazon, and Yelp drop out because their high composite Index came from the take side.

Take-only produces the biggest ranking shifts. GitHub tops the list at take 9. Mistral moves from 28th to 6th. OpenAI and Google reach 4th and 5th.

Across all five weightings, Adobe is the only company in the top-5 under every scheme. Amazon, Perplexity, Reddit, and Yelp are in the top-10 under all five. The core of the leaderboard holds. The rest of the ordering depends on whether the forbid or take side is weighted more heavily.

What the numbers add up to

Terms of Service and robots.txt are two independent signals about a site’s scraping stance. In this sample, they agree at the extremes and diverge in the middle.

At the strict end, 6 companies write both a strong Terms clause and a robots.txt block against AI training crawlers. X Corp, Reddit, Amazon, Yelp, Meta, and ByteDance.

At the permissive end, 14 companies write neither. Anthropic, Microsoft, Apple, and seven others score 0 on the forbid component, Atlassian, Common Crawl, Zoom, and Booking stay below 2.5, and none of the 14 blocks an AI training crawler in robots.txt.

Between the two extremes, 15 companies split the signal. 13 declare a stance in Terms but do nothing in robots.txt (paper-only forbidders). 2 block in robots.txt without a matching Terms clause (silent enforcers).

The Terms themselves are mostly copies of shared legal templates. 17 sites use “robot, spider, scraper” phrasing. 13 use “data mining, robots or similar data gathering.” 12 share anti-circumvention wording. The double standard is largely the result of anti-scraping boilerplate being adopted by default from industry templates, while extraction clauses are drafted separately for each company’s business model.

The Index isolates the gap between those two sides. Under every alternative weighting, Adobe stays in the top-5. Amazon, Perplexity, Reddit, and Yelp stay in the top-10. The rest of the ranking depends on which side is weighted more heavily.

The data describes the gap between what a company’s own Terms forbid and what its Privacy Policy claims about data collected from elsewhere. It doesn’t describe what a court would ultimately allow, only what the companies write in their own public documents.

Conclusion

The 402 verified clauses in this study describe what companies write, not what any court would ultimately enforce. Regulation in the US and EU is still working through the same questions. The July 2026 snapshot is a reference point for measuring how the gap changes as those decisions arrive.

Cite this study as HasData, The AI Hypocrisy Index, https://hasdata.com/blog/ai-hypocrisy-index. The verified clauses and scores behind it are archived on Zenodo under DOI 10.5281/zenodo.23499320.