Web Scraping Without Getting Blocked

A scraper gets blocked when the site can tell it apart from a browser, and it can do that on four layers. The TLS handshake and HTTP/2 settings give away the HTTP library before the first header arrives. The IP address gives away the datacenter and the request rate. The header set and the JavaScript-visible browser properties give away a headless browser or a mismatched User-Agent. Request timing, hidden links, and ignored robots.txt rules give away the crawler.

The twelve techniques below are what web scraping without getting blocked takes in practice, ordered from the layer that catches a scraper first to the one that catches it last, and every technique comes with Python code and the output it returns. The fingerprint numbers in the first section come from requests 2.34, curl_cffi 0.16, and Chrome 151 hitting the same test server.

Why Sites Block Scrapers

A block shows up as a status code and a page you did not ask for. The response is a 403 Forbidden with an error page, a 429 Too Many Requests after a burst of requests, a 503 that carries a JavaScript challenge instead of the page, or a 200 whose body is a challenge form or an empty shell. What triggered it is one of four checks.

1. The connection gives the client away first. Every TLS client announces its cipher suites, extensions, and curves in the ClientHello, every HTTP/2 client announces its SETTINGS frame, and anti-bot systems hash both into fingerprints (JA3, JA4, an Akamai-style HTTP/2 hash) that they compare with those of real browsers. A Python HTTP library has its own fingerprint, and no header changes it.
2. The address is scored next. Datacenter IP ranges, addresses already flagged as proxies, too many requests from one address, and a visitor from a country the site never serves all lower the score before the request body is read.
3. The browser is checked for contradictions. Headers and client hints that do not match each other, a User-Agent that says Chrome while the TLS fingerprint says Python, navigator.webdriver set to true, a screen size that does not exist, and a timezone that disagrees with the IP all count.
4. The behaviour is judged over time. Requests every 2.000 seconds, forty pages in the order of the sitemap, no images or scripts ever requested, a click on a link that is invisible to humans, and pages that robots.txt asked not to touch.

Cloudflare, DataDome, and HUMAN combine all four, and the section after the techniques explains how each of them does it. The techniques themselves work against any of the three.

1. Match the TLS and HTTP/2 Fingerprint

The TLS fingerprint is checked before the User-Agent, so it comes first. JA3 takes the TLS version, cipher list, extension list, elliptic curves, and point formats from the ClientHello and hashes them into one MD5. JA4 does the same with sorted extensions and keeps a readable prefix. t13d1516h2 means TLS 1.3, a domain in the SNI field, 15 ciphers, 16 extensions, and h2 as the first ALPN value. Cloudflare’s documentation notes that JA4 sorts the extensions, which cuts the number of distinct fingerprints modern browsers produce.

requests speaks HTTP/1.1 through OpenSSL, so the server sees a fingerprint that no browser has ever produced, and no header changes that. The tool that fixes it is curl_cffi, a Python binding for a curl build that reproduces the TLS and HTTP/2 handshake of a chosen browser. Install it with pip install curl_cffi and pass impersonate="chrome", or one of the 54 targets such as safari, firefox, edge, and versioned ones like chrome136 or safari184_ios.

The script below sends three requests to a TLS test endpoint and prints what the server saw. The third request repeats the second one to show what changes between connections.

import requests
from curl_cffi import requests as cffi_requests

URL = "https://tls.browserleaks.com/json"

plain = requests.get(URL, timeout=30).json()
chrome = cffi_requests.get(URL, impersonate="chrome", timeout=30).json()
chrome_again = cffi_requests.get(URL, impersonate="chrome", timeout=30).json()

for name, data in (("requests", plain), ("curl_cffi chrome", chrome), ("curl_cffi chrome, 2nd run", chrome_again)):
    print(f"{name:26} JA3 {data['ja3_hash']}  JA4 {data['ja4']}  HTTP/2 {data['akamai_hash'] or '-'}")
    print(f"{'':26} UA  {data['user_agent']}")

The output from one run, with a fourth row added from a real Chrome 151 driven by Playwright against the same endpoint (the code for that is in technique 7).

The requests fingerprint says h1 and 17 ciphers, which no current browser sends, so a site that keys on JA4 can reject it regardless of the User-Agent header. The impersonated JA4 and HTTP/2 hash are identical to the real browser’s, while JA3 differs between the two curl_cffi runs. Chrome randomizes its extension order on every connection, curl_cffi reproduces that, and JA3 (unsorted) changes with it while JA4 (sorted) stays put. A block list built on JA3 hashes is therefore weaker than one built on JA4, and both are blind to curl_cffi.

The impersonation target also sets the User-Agent and the other default headers to match, in this case a Chrome 150 on macOS. Leave them as they are unless you change the whole set consistently. A Windows User-Agent on top of a macOS TLS profile adds a mismatch instead of removing one.

2. Send a Full Browser Header Set

Once the connection looks like Chrome, the headers have to look like Chrome too, and that means the whole set in the browser’s order, not a single User-Agent string pasted over python-requests/2.34.2. Chrome sends Accept, Accept-Language, Accept-Encoding, the Sec-Ch-Ua client hints with brand and platform, the Sec-Fetch-* headers that describe the navigation, Upgrade-Insecure-Requests, and a Referer when it followed a link.

The values have to agree with each other. Sec-Ch-Ua-Platform says "Windows" when the User-Agent says Windows NT, the brand versions match the version in the User-Agent, and the Referer is a page that plausibly links to the target (a search engine result page is the usual choice). Image and file hosts check the Referer too, so a request for an image without the page that embeds it as Referer gets a 403 from hotlink protection while the same request with it passes. The safest source is DevTools in the browser you are imitating, Network tab, any document request, Headers panel.

In requests, assigning a dict to session.headers replaces the library defaults, so the order you write is the order that goes on the wire. Chrome also advertises br and zstd in Accept-Encoding. Add them only with the brotli and zstandard packages installed, otherwise the body arrives compressed and undecoded.

import random
import requests

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
]


def browser_headers(user_agent: str) -> dict:
    platform = '"Windows"' if "Windows" in user_agent else '"macOS"'
    return {
        "User-Agent": user_agent,
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate",
        "Referer": "https://www.google.com/",
        "Sec-Ch-Ua": '"Not=A?Brand";v="99", "Google Chrome";v="151", "Chromium";v="151"',
        "Sec-Ch-Ua-Mobile": "?0",
        "Sec-Ch-Ua-Platform": platform,
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "cross-site",
        "Upgrade-Insecure-Requests": "1",
    }


session = requests.Session()
session.headers = browser_headers(random.choice(USER_AGENTS))  # replaces the python-requests defaults, order included

response = session.get("https://httpbin.org/headers", timeout=30)
print(response.status_code, response.json()["headers"]["User-Agent"])

Rotate User-Agents from a short list of current browsers rather than a thousand-line file scraped from the web. Old versions and rare browsers are a small share of real traffic, and a small share is what the models flag, so one boring, current Chrome does better than variety, and every User-Agent in the list needs its own matching client hints.

3. Rotate IP Addresses and Match the Geo

The address layer counts requests per IP and checks where the IP comes from. DataDome’s documentation lists reputational detection as one of its four model groups, with addresses “identified as a data center or residential proxy” by its models, so a datacenter range is a mark against the request before anything else is read. The answer is a pool of proxies for web scraping that spreads requests over many addresses, residential or mobile for protected sites, from the country the site serves. A German retailer visited from forty Vietnamese addresses in one afternoon is not a pattern real customers produce.

Rotation needs a retirement rule as well. When an address starts returning 403 or 429 it is burned for that site, and sending it again wastes requests and teaches the anti-bot system what your pool looks like. The function below picks a random live proxy, retires one on the first block, and raises once the pool is exhausted. The pool comes from an environment variable so that credentials stay out of the code.

import os
import random
import requests

# one "http://user:pass@host:port" per proxy; an empty pool means a direct connection
PROXY_POOL = [p for p in os.environ.get("PROXY_POOL", "").split(",") if p]
RETIRED = set()


def fetch(url: str) -> requests.Response:
    while True:
        live = [p for p in PROXY_POOL if p not in RETIRED]
        if PROXY_POOL and not live:
            raise RuntimeError("every proxy in the pool is blocked")
        proxy = random.choice(live) if live else None
        proxies = {"http": proxy, "https": proxy} if proxy else None
        response = requests.get(url, proxies=proxies, timeout=30)
        if response.status_code in (403, 429) and proxy:
            RETIRED.add(proxy)  # this address is burned for this site, do not send it again
            continue
        return response


for _ in range(3):
    print(fetch("https://httpbin.org/ip").json()["origin"])

Sticky sessions matter as much as rotation. A login, a cart, or a multi-page form has to stay on one address for its whole life, so the pool needs a per-session proxy as well as a per-request one, and requests takes authenticated and SOCKS proxies through the same proxies argument.

4. Randomize Request Timing

A request every two seconds, on the dot, for six hours, is a machine, and the interval is the most basic behavioural signal a rate limiter sees. Real visitors read, scroll, and leave. Random delays between pages, a few seconds wide, and a low concurrency per domain keep the rate inside what a small group of humans would generate.

import random
import time
import requests

session = requests.Session()
session.headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"

for page in range(1, 4):
    response = session.get(f"https://httpbin.org/anything/catalog?page={page}", timeout=30)
    print(response.status_code, response.json()["url"])
    time.sleep(random.uniform(2, 6))  # a person does not open the next page every 2.000 seconds

The order of pages is a signal too. A crawler that walks a category listing page by page and never opens a product is visible in the access log without any fingerprinting, so mix detail pages into the listing walk when the target is protected.

5. Back Off on 429 and 503

A 429 means the site counted your requests and asked you to slow down, and the response often carries a Retry-After header with the number of seconds to wait. Cloudflare’s rate limiting shows the same thing as error 1015 inside a 429 page. A 503 from an anti-bot layer can be a challenge page rather than an outage, so the same backoff applies. Ignoring either and retrying immediately turns a temporary limit into a ban.

import time
import requests


def get_with_backoff(session: requests.Session, url: str, retries: int = 3) -> requests.Response:
    delay = 2.0
    for attempt in range(1, retries + 1):
        response = session.get(url, timeout=30)
        if response.status_code not in (429, 503):
            return response
        retry_after = response.headers.get("Retry-After")
        wait = float(retry_after) if retry_after and retry_after.isdigit() else delay
        print(f"attempt {attempt}: {response.status_code}, waiting {wait:.0f}s")
        time.sleep(wait)
        delay *= 2
    response.raise_for_status()
    return response


with requests.Session() as session:
    get_with_backoff(session, "https://httpbin.org/status/429")

Against an endpoint that always answers 429 the function waits 2, 4, and 8 seconds and then raises HTTPError.

attempt 1: 429, waiting 2s
attempt 2: 429, waiting 4s
attempt 3: 429, waiting 8s

Giving up there is the point. After three refusals the right move is a longer pause or a different address rather than a fourth attempt, and retries with jitter on connection errors follow the same shape.

6. Keep One Session with Its Cookies

Anti-bot systems hand out cookies after they have checked a client. Cloudflare stores the result of its JavaScript Detections in a cf_clearance cookie and repeats the detection every 15 minutes, DataDome sets a datadome cookie that lives a year and is required for its checks to work, and HUMAN sends its risk score back in a _px3 cookie, with _pxhd for server-side detection. A scraper that drops cookies between requests takes the full check every time, which is the behaviour those systems are built to notice. Cloudflare’s documentation also states that a challenge solved from a different IP than the one that received it does not count, so a session has to keep its address as well as its cookies.

import requests

with requests.Session() as session:
    session.headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"
    session.get("https://httpbin.org/cookies/set?session_id=abc123", timeout=30)  # the server sets a cookie
    response = session.get("https://httpbin.org/cookies", timeout=30)  # the same session sends it back
    print(response.json())

The second request carries the cookie the first one received.

{'cookies': {'session_id': 'abc123'}}

One session per identity is the rule. The same cookie jar behind two IP addresses, or two User-Agents sharing one jar, contradicts itself, and contradiction is what the scoring engines look for.

7. Use a Headless Browser Where the Page Needs JavaScript

When the content is rendered client-side or the site serves a JavaScript challenge, an HTTP client cannot get past the first response, and a real browser can. Playwright drives Chromium, Firefox, and WebKit from Python, and the browser brings its own, correct TLS and HTTP/2 fingerprint. Install it with pip install playwright followed by playwright install chromium. The catch is what the default headless mode reveals about itself.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://tls.browserleaks.com/json", wait_until="domcontentloaded")
    data = page.evaluate("() => JSON.parse(document.body.innerText)")
    print("UA:       ", page.evaluate("navigator.userAgent"))
    print("webdriver:", page.evaluate("navigator.webdriver"))
    print("JA4:      ", data["ja4"])
    browser.close()

The TLS side passes, the browser side does not.

UA:        Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/151.0.7922.34 Safari/537.36
webdriver: True
JA4:       t13d1516h2_8daaf6152771_806a8c22fdea

HeadlessChrome in the User-Agent and navigator.webdriver set to true are the signals Cloudflare’s JavaScript Detections engine is built around, its documentation describes the engine as catching headless browsers and other automation tools. Selenium shows the same automation banner in a visible window, and the same flags in a headless one.

Both flags come off together with the rest of the browser profile in the next technique.

8. Keep the Browser Fingerprint Consistent

The JavaScript that anti-bot systems inject collects far more than the User-Agent. DataDome’s tag records mouse movements and key strokes and data about the OS, the browser, and the GPU, and HUMAN’s sensor collects “hundreds of features” that its detector compares with a library of bot profiles. Randomizing those values makes a fingerprint that exists nowhere else, which is worse than a common one. Consistency is what passes. The User-Agent, the client hints, navigator.platform, the locale, the timezone, and the viewport have to describe one plausible machine, and the IP address has to come from where that machine claims to be.

Launching the installed Google Chrome (channel="chrome") instead of the bundled Chromium makes the client hints say “Google Chrome”, which matches a current Chrome User-Agent. The bundled build reports only “Chromium” and contradicts the string. Overriding the User-Agent removes the HeadlessChrome token, and it has to carry the same major version as the browser that is really running. The AutomationControlled flag turns navigator.webdriver off.

from playwright.sync_api import sync_playwright

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"

with sync_playwright() as p:
    browser = p.chromium.launch(
        channel="chrome",  # the installed Google Chrome, so the client hints say "Google Chrome" like the UA does
        headless=True,
        args=["--disable-blink-features=AutomationControlled"],  # navigator.webdriver becomes false
    )
    context = browser.new_context(
        user_agent=UA,  # same major version as the installed Chrome, without the HeadlessChrome token
        locale="en-US",
        timezone_id="America/New_York",
        viewport={"width": 1366, "height": 768},
    )
    page = context.new_page()
    page.goto("https://example.com", wait_until="domcontentloaded")
    print(page.evaluate("""() => ({
        userAgent: navigator.userAgent,
        brands: navigator.userAgentData.brands.map(b => `${b.brand} ${b.version}`).join(", "),
        platform: navigator.platform,
        webdriver: navigator.webdriver,
        language: navigator.language,
        timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
    })"""))
    browser.close()

Everything the page can read now tells one story, a Chrome 151 on Windows in New York.

{'userAgent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36', 'brands': 'Not=A?Brand 99, Google Chrome 151, Chromium 151', 'platform': 'Win32', 'webdriver': False, 'language': 'en-US', 'timezone': 'America/New_York'}

The proxy for this session should then be a US residential address, and the Accept-Language header en-US. Canvas and WebGL hashes stay whatever the real GPU produces, and those are common values shared by every machine with that hardware. Fingerprint checkers such as browserleaks.com or the EFF’s Cover Your Tracks show what a page sees, and they are the place to test a profile before pointing it at a target.

9. Skip Honeypot Links

A honeypot is a link no human can click. It is hidden with display: none, visibility: hidden, opacity: 0, a class like hidden, or a position far off screen, and it leads to a URL that only a crawler following every href would ever request. One hit marks the client, and the site can then block the session or the IP. The check is a few lines of beautifulsoup4 (pip install beautifulsoup4). Before following a link, walk up its ancestors and drop it if anything on the way is hidden.

import re
from bs4 import BeautifulSoup

html = """
<a href="/products">Products</a>
<a href="/trap-1" style="display:none">Special offer</a>
<a href="/trap-2" style="visibility: hidden">Deals</a>
<a href="/trap-3" class="hidden">Archive</a>
<div style="display: none"><a href="/trap-4">Old catalog</a></div>
<a href="/about">About</a>
"""

HIDDEN_STYLE = re.compile(r"display\s*:\s*none|visibility\s*:\s*hidden|opacity\s*:\s*0(?![.\d])", re.I)


def is_visible(tag) -> bool:
    for node in (tag, *tag.parents):
        if node.name in (None, "[document]"):
            continue
        if HIDDEN_STYLE.search(node.get("style", "")) or "hidden" in node.get("class", []):
            return False
    return True


soup = BeautifulSoup(html, "html.parser")
links = [a["href"] for a in soup.find_all("a", href=True) if is_visible(a)]
print(links)

Four traps out, two real links left.

['/products', '/about']

Inline styles are the easy case. Classes defined in an external stylesheet need the CSS parsed as well, and a headless browser resolves that for free, since element.checkVisibility() in the page returns the rendered answer.

10. Respect robots.txt and the Site’s Off-Peak Hours

robots.txt is the site telling you what it does not want crawled and, through Crawl-delay, how fast it is willing to be crawled. Reading it costs one request and avoids the paths most likely to be trapped or rate-limited. The standard library parses it. The second half of the snippet handles the other timing signal, load. A crawl that runs while the site’s own customers are shopping slows the site down and stands out in the logs, so schedule it for the site’s night, in the site’s timezone, not your server’s. On Windows, zoneinfo needs the tzdata package (pip install tzdata).

import urllib.robotparser
from datetime import datetime
from zoneinfo import ZoneInfo

robots = urllib.robotparser.RobotFileParser("https://www.python.org/robots.txt")
robots.read()
print("fetch allowed:", robots.can_fetch("*", "https://www.python.org/downloads/"))
print("crawl-delay:  ", robots.crawl_delay("*"))

site_time = datetime.now(ZoneInfo("America/Los_Angeles"))  # the site's audience, not your server
if 1 <= site_time.hour < 6:
    print(f"{site_time:%H:%M} at the site: off-peak, run the full crawl")
else:
    print(f"{site_time:%H:%M} at the site: peak hours, keep the rate low")

A fixed start time is a pattern of its own. A job that fires at 02:00 every night is easy to spot in a week of logs, so let the start drift by a random half hour.

11. Use the Site’s Own Data Endpoints

Many pages are shells that fetch their data from a JSON endpoint after load, and that endpoint is often less protected than the HTML and cheaper to parse. DevTools, Network tab, filter Fetch/XHR, reload the page, and read the requests the page makes for itself. The Response tab shows the JSON the page received, the Preview tab shows the same data as a tree, and the Headers tab shows what the page sent to get it, which is what your request needs to send too.

The screenshot is the quotes.toscrape.com practice site, whose infinite scroll fetches quotes?page=N and gets the quotes as JSON with a has_next flag. The endpoint Chrome’s address bar calls for search suggestions is another small example of the pattern. It returns a JSON array, takes the same query parameters the browser sends, and needs nothing beyond a browser User-Agent.

import json
import requests

# the endpoint Chrome's address bar calls for suggestions, visible in DevTools > Network > Fetch/XHR
response = requests.get(
    "https://www.google.com/complete/search",
    params={"q": "web scraping", "client": "chrome", "hl": "en", "gl": "us"},
    headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"},
    timeout=30,
)
query, suggestions, *_ = json.loads(response.text)
print(suggestions[:5])

The first five suggestions for the query, straight from the array.

['web scraping python', 'web scraping meaning', 'web scraping ai', 'web scraping services', 'web scraping vs web crawling']

Real endpoints usually want more than a User-Agent. Copy the Referer, Origin, X-Requested-With, and any token header from the browser’s request, and expect the token to expire. Mobile apps talk to the same kind of endpoints, and an intercepting proxy such as mitmproxy on the phone’s Wi-Fi shows them, though app traffic tends to carry signed requests and certificate pinning that make this the harder route.

12. Hand the Blocking Layer to a Scraping API

Everything above has to be maintained. Browser versions move every four weeks, curl_cffi targets follow them, proxy pools burn addresses, and a site that switches to a new anti-bot vendor breaks a working setup overnight. At some volume that upkeep costs more than the data. Our Web Scraping API takes the connection and address layers off your hands. Each request goes out through a datacenter or residential proxy in the country you choose, with optional JavaScript rendering and your own headers and cookies, and only successful responses are billed.

import os
import re

import requests

response = requests.post(
    "https://api.hasdata.com/scrape/web",
    headers={"x-api-key": os.environ["HASDATA_API_KEY"], "Content-Type": "application/json"},
    json={
        "url": "https://books.toscrape.com/",
        "proxyType": "residential",
        "proxyCountry": "US",
        "jsRendering": False,
    },
    timeout=60,
)
print(response.status_code)
data = response.json()
print(sorted(data.keys()))
print(re.search(r"<title>(.*?)</title>", data["content"], re.S).group(1).strip())

The request went out through a US residential proxy, and the response wraps the page with its status and headers.

200
['content', 'headers', 'requestMetadata', 'statusCode', 'statusText']
All products | Books to Scrape - Sandbox

This call costs five credits (residential proxy, no rendering). With jsRendering set to true it costs fifteen, and a datacenter proxy without rendering costs one. The techniques in sections 1 through 11 still apply to what you do with the page afterwards, but the fingerprint, the pool, and the retries stop being your code.

How Cloudflare, DataDome, and HUMAN Detect Bots

The three systems behind most block pages share the four layers and differ in where the decision is made and what the challenge looks like. Everything below comes from their own documentation.

Cloudflare gives every request a bot score from 1 to 99, where 1 means certainly automated. A heuristics engine matches requests against a database of known malicious fingerprints and scores deterministic hits at 1, a machine-learning engine trained on billions of daily requests produces most of the other scores, and JavaScript Detections injects an invisible script from /cdn-cgi/challenge-platform/ that identifies headless browsers and writes its verdict into the cf_clearance cookie. JA3 and JA4 fingerprints are part of the Enterprise Bot Management product. What a scraper sees is a 403 with error 1010 when the browser signature fails the Browser Integrity Check, a 429 with error 1015 from a rate limiting rule, a 1020 from a firewall rule, or a Managed Challenge page.

DataDome runs its models in four groups. Signature-based detection uses TLS fingerprints, browser fingerprints, and HTTP headers, behavioural detection catches aggressive or repetitive patterns, reputational detection scores the IP address (including whether its models have flagged it as a datacenter or residential proxy), and a scanner-detection group watches for probing of predictable paths. The client side is a JavaScript tag that reports mouse movements, key strokes, and OS, browser, and GPU data, and it needs read and write access to the datadome cookie. Its responses escalate from Device Check, an invisible verification, to the Slider, its own CAPTCHA, to a hard block, and its analytics classify a client as Blocked when it “detected an automated challenge solver or a challenge-solving service”. That last line is what happens to CAPTCHA-solving services against DataDome. The solver may pass the puzzle, the session is still flagged.

HUMAN, which merged with PerimeterX and kept its architecture, splits the work into three parts. A Sensor script on the page collects hundreds of signals about the device, the browser, and the user’s interaction, a cloud Detector turns them into a risk score per request and sends it back in an encrypted cookie (_px3, with _pxhd for server-side detection), and an Enforcer module in the CDN or application applies the allow, block, or challenge decision. Its HUMAN Challenge is the press-and-hold widget, built to be hard to solve through API calls, automation, or CAPTCHA farms, and to record how solvers behave.

All three score the request on several signals at once. A perfect TLS fingerprint with a headless User-Agent, or a consistent browser on a datacenter IP, still scores badly.

Detection Mechanisms and Countermeasures

Each row names a check from the sections above, the response it produces, and the technique that answers it.

Rows one through three decide most outcomes before the request body is read, and that order is the order of the techniques.

Conclusion and Takeaways

The fingerprint table in the first section is the whole argument in four rows. A Python HTTP client announces itself in the TLS handshake, so header tricks alone stop nothing on a site that checks JA4, and curl_cffi or a real browser has to come first. After that the address decides, then the browser profile, then behaviour. A scraper that passes the first three layers and then requests forty pages in forty seconds from one IP still ends up with a 429, and one that drops its cookies takes the challenge on every request.

Cloudflare, DataDome, and HUMAN publish enough about their detection to know what they look at, and each of them scores several layers at once, so a fix on one layer buys nothing without the others. For a handful of sites the twelve techniques above are an afternoon of work. For a fleet of them, moving the connection and address layers to a scraping API is usually cheaper than keeping that afternoon’s work current.