Web Crawling with Python in 2026 (Benchmarked)

Every Python crawler tutorial recommends a different tool, from Requests + BeautifulSoup for a first script to Scrapy for scale, Playwright for JavaScript-heavy sites, and Crawlee for modern workflows. The tradeoffs between them are real but rarely measured, so the choice usually comes down to whichever guide you land on first.

I benchmarked all four on the same 100-page crawl of books.toscrape.com and got numbers that flip the common advice. Scrapy runs 4.1x faster than Requests + BeautifulSoup. Crawlee for Python, marketed for scale, is roughly the same speed as sync Requests because its default HTTP client does browser fingerprinting on every request. httpx.AsyncClient hits 80% of Scrapy’s throughput with a tenth of the setup.

The minimal Python crawler is under 20 lines.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse

BASE = "https://books.toscrape.com/"
visited, queue = set(), [BASE]

while queue and len(visited) < 20:
    url = queue.pop(0)
    if url in visited:
        continue
    resp = requests.get(url, timeout=10)
    if resp.status_code != 200:
        continue
    visited.add(url)
    soup = BeautifulSoup(resp.text, "html.parser")
    for a in soup.find_all("a", href=True):
        link = urljoin(url, a["href"]).split("#")[0]
        if urlparse(link).netloc == urlparse(BASE).netloc and link not in visited:
            queue.append(link)

print(f"Crawled {len(visited)} pages")

That’s the foundation for every approach in this article. Change one line and the same code becomes depth-first. Wrap it in httpx.AsyncClient and it runs 3x faster. Plug it into Scrapy and you get retries, middleware, and rate limiting for free. When each of those steps earns its complexity is the whole question.

Key numbers:

• Scrapy is the fastest general-purpose Python crawler on static sites (14.8 pages/sec in the benchmark, 4.1x Requests + BeautifulSoup). Use it for crawls over a few hundred pages.
• Crawlee for Python is not faster than sync Requests. Its default HTTP client trades throughput for anti-bot fingerprinting. Only worth it when the target actively blocks default clients.
• BFS crawls all top-level sections first, DFS chases one link chain deep. Same code, queue.pop(0) vs queue.pop(-1). Pick the wrong one and 50 pages of budget hit zero of your target pages.
• Playwright is 2x slower than Requests on the same URL, but on JavaScript-rendered pages it retrieves 54% more content. The 2x penalty is the price of real data instead of an empty HTML shell.

You need Python 3.11+ and comfort with async/await. No prior Scrapy or Crawlee experience required.

What is a Python web crawler

A web crawler starts from one URL, downloads the page, extracts links, and repeats the process on those new URLs until it hits a stop condition. A scraper extracts data from a page you already have. Same tools, different scope. A crawler is a scraper wrapped in a URL discovery loop, and that URL-discovery loop is what the rest of this article measures across four different Python tools.

Every crawler runs the same loop.

Every tool in this article, from Requests + BeautifulSoup to httpx.AsyncClient, Scrapy, Crawlee, and Playwright, implements this same loop. The difference is which middleware each one wraps around the loop, whether that’s retry logic, rate limiting, header management, proxy rotation, or browser rendering.

SEO site audits walk every page on a domain and log status codes, canonical tags, and broken internal links. Price monitoring enumerates product pages across a catalog daily and diffs prices against yesterday. Dataset building collects every article, job post, or product listing into structured storage (CSV, JSON, database). Link graph analysis builds a directed graph of internal links to find orphan pages, hub pages, and crawl depth from the homepage. These are what production crawlers do, and picking the right tool comes down to how many pages, how often, and whether the target requires JavaScript to render its content.

Choosing the right Python crawler tool

The tool for a Python crawler is set by two things, the target’s shape (static, dynamic, or blocked) and the crawl’s scale (from a few dozen pages to hundreds of thousands). The table below compares five Python crawler tools across target size, JavaScript support, anti-bot handling, setup complexity, and typical use case.

Every column has an escape hatch. Scrapy can drive Playwright through scrapy-playwright for JS-heavy targets, Crawlee has both HTTP and browser crawler variants, and httpx.AsyncClient can be pointed at a rotating proxy for basic anti-bot. The table shows what each tool does out of the box.

Requests + BeautifulSoup is the crawler you can write in 20 lines with no dependencies beyond pip install requests beautifulsoup4. Zero framework overhead means fastest iteration for small crawls, at 3.6 pages/sec on the benchmark. It hits a wall around a few hundred pages because sync HTTP blocks on every fetch. Retries, rate limiting, and middleware are what you write yourself.

httpx.AsyncClient brings async concurrency to the same loop. asyncio.gather with concurrency=5 hit 11.5 pages/sec on the benchmark, a 3.2x speedup over sync Requests. You still write the queue, the visited set, and the retry logic, but the fetch part scales linearly with concurrency. Right choice when the crawl outgrows sync Requests but doesn’t need Scrapy.

Scrapy adds a scheduler, AutoThrottle for polite crawling, retry middleware, pipelines for data output, and dozens of contrib packages on top of the same loop. Setup is one scrapy startproject command and a spider file. On the benchmark it topped 14.8 pages/sec (4.1x sync Requests). Use it when the crawler is production infrastructure. The tradeoff is project structure, because a Scrapy spider doesn’t run as a single script.

Crawlee for Python targets anti-bot scenarios where default HTTP clients get blocked. Its default HTTP client (impit) performs browser TLS fingerprinting on every request, which gets through defenses that reject python-requests User-Agents outright. The throughput cost is real, at 3.9 pages/sec on the benchmark. Worth it when the target is Cloudflare-protected, Datadome-protected, or similar. Overkill when the target has no defenses.

Playwright launches Chromium, runs the JavaScript, and returns the fully-rendered DOM. On the benchmark, Playwright fetched an SPA in 2.7s vs 1.2s for Requests, 2.2x slower, but retrieved 54% more content because Requests returned only the empty shell. Use it for client-rendered content and SPAs where JS execution is required. Combine it with Scrapy through the scrapy-playwright package or Crawlee’s native Playwright integration when the target has both JavaScript and anti-bot defenses.

Building your first Python crawler (Requests + BeautifulSoup)

The 20-line crawler from the intro leaves out robots.txt compliance, rate limiting, retry on transient errors, and a configurable page cap. The version below includes all four.

"""Basic Python crawler with robots.txt, rate limit, and retry."""
import time
import random
from urllib.parse import urljoin, urlparse
from urllib.robotparser import RobotFileParser
import requests
from bs4 import BeautifulSoup

BASE = "https://books.toscrape.com/"
MAX_PAGES = 50
DELAY = 0.5           # seconds between requests
MAX_RETRIES = 3
UA = "sample-python-crawler/1.0"

rp = RobotFileParser()
rp.set_url(urljoin(BASE, "/robots.txt"))
rp.read()

session = requests.Session()
session.headers["User-Agent"] = UA

def fetch(url):
    for attempt in range(MAX_RETRIES):
        try:
            resp = session.get(url, timeout=10)
            if resp.status_code in (429, 500, 502, 503, 504):
                raise requests.HTTPError(f"HTTP {resp.status_code}")
            return resp
        except (requests.ConnectionError, requests.Timeout, requests.HTTPError):
            if attempt == MAX_RETRIES - 1:
                return None
            time.sleep((2 ** attempt) + random.random())

visited, queue = set(), [BASE]
base_domain = urlparse(BASE).netloc

while queue and len(visited) < MAX_PAGES:
    url = queue.pop(0)
    if url in visited or not rp.can_fetch(UA, url):
        continue
    resp = fetch(url)
    if resp is None or resp.status_code != 200:
        continue
    visited.add(url)
    soup = BeautifulSoup(resp.text, "html.parser")
    for a in soup.find_all("a", href=True):
        link = urljoin(url, a["href"]).split("#")[0]
        if urlparse(link).netloc == base_domain and link not in visited and link not in queue:
            queue.append(link)
    time.sleep(DELAY)

print(f"Crawled {len(visited)} pages")

Python’s built-in urllib.robotparser reads the target’s robots.txt once at startup, and .can_fetch(UA, url) filters every candidate URL before the network call. The fetch function retries on 429, 5xx, and network errors with exponential backoff plus jitter, capped at 3 attempts. time.sleep(DELAY) after each successful fetch caps the request rate at 2 per second. MAX_PAGES lives at the top of the file for easy tuning.

The link not in queue check is linear in queue length. It’s fine at small scale, but a crawl of 100,000+ URLs slows down noticeably, and the in-memory frontier itself starts eating RAM at that size. A larger crawl needs an on-disk frontier and a bloom filter for dedup.

BFS versus DFS in one line

The crawler above uses queue.pop(0) to pick the next URL to visit. That FIFO queue spreads the crawl out from the seed level by level, which is what breadth-first search (BFS) means. Change the index to queue.pop(-1) and the queue turns into a LIFO stack. The crawler then picks the most recently discovered URL and follows it as deep as possible before backtracking, which is depth-first search (DFS).

# BFS: FIFO queue, pop from front, spreads level by level
url = queue.pop(0)

# DFS: LIFO stack, pop from back, dives down one branch
url = queue.pop(-1)

Consider a small site with a home page, three category pages, and two leaf pages under each category. The diagram below shows the visit order for each strategy on this ten-node tree.

The numbers on each node show the visit order. BFS fills the tree one level at a time (1, then 2-3-4, then 5-10). DFS goes deep into one branch before moving on (1, then 2, then 3-4, then jumps to the next branch).

On a 50-page crawl of books.toscrape.com, BFS visited 48 category pages at depth 1 and no book detail pages. DFS followed a single pagination chain to depth 49 and visited no category pages. The URL sets from the two runs do not overlap.

For full-site audits, price monitoring, and sitemap builds, BFS is better because it hits every top-level section before diving. DFS fits when a single chain is the goal, like a forum thread walk or paginated enumeration.

Storing extracted data

Printing URL counts is fine for a smoke test. Actual crawlers extract fields from each page and write them to disk. CSV, JSON, and SQLite each solve a slightly different case.

CSV works when the extracted data has a fixed schema that fits a spreadsheet. csv.DictWriter handles quoting and header rows.

import csv

rows = [
    {"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
     "title": "A Light in the Attic", "price": 51.77},
]

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["url", "title", "price"])
    writer.writeheader()
    writer.writerows(rows)

JSON fits nested or variable-schema data. Newline-delimited JSON (NDJSON) writes one object per line and streams without holding all rows in memory.

import json

rows = [
    {"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
     "title": "A Light in the Attic", "price": 51.77, "categories": ["Poetry"]},
]

with open("books.ndjson", "w", encoding="utf-8") as f:
    for row in rows:
        f.write(json.dumps(row) + "\n")

SQLite fits anything that outgrows CSV or NDJSON, especially when you need indexed lookups or want to resume a crawl. Batch inserts inside a single transaction are dramatically faster than one commit per row.

import sqlite3

rows = [
    ("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
     "A Light in the Attic", 51.77),
]

conn = sqlite3.connect("crawl.db")
conn.execute("""
    CREATE TABLE IF NOT EXISTS pages (
        url TEXT PRIMARY KEY,
        title TEXT,
        price REAL,
        crawled_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
    )
""")
with conn:
    conn.executemany(
        "INSERT OR REPLACE INTO pages(url, title, price) VALUES (?, ?, ?)",
        rows,
    )
conn.close()

The SQLite version doubles as crawl state. Restarting after a crash can query SELECT url FROM pages and skip already-visited URLs.

URL frontier and deduplication at scale

In-memory set() and list handle URL counts up to a few thousand comfortably. At 10,000 URLs the visited set fits in a few MB. Growth is linear from there, reaching several hundred MB total at a million URLs. On top of that, the link not in queue check scans the entire queue on every new discovered link, and the queue keeps growing. The crawler slows page by page.

The frontier and visited state both live in RAM and grow with discovered URLs. Neither survives a crash. Restarting a 500,000-page crawl from scratch after a failure at page 250,000 wastes hours of network work.

The fix moves the frontier and visited state to disk, uses an indexed lookup that stays fast as the URL count grows, and lets the crawler resume from where it stopped.

SQLite-backed frontier

SQLite handles both jobs at once. A single table stores discovered URLs with their state (queued or visited), the database file survives crashes, and the primary key index on url keeps lookups fast even at a million rows, unlike scanning a Python list.

"""Crawler with SQLite-backed frontier for resumable large crawls."""
import sqlite3
from urllib.parse import urljoin, urlparse
import requests
from bs4 import BeautifulSoup

BASE = "https://books.toscrape.com/"
MAX_PAGES = 100
DB_PATH = "crawl.db"

session = requests.Session()
session.headers["User-Agent"] = "sample-python-crawler/1.0"

conn = sqlite3.connect(DB_PATH)
conn.execute("""
    CREATE TABLE IF NOT EXISTS frontier (
        url TEXT PRIMARY KEY,
        visited INTEGER DEFAULT 0
    )
""")

def enqueue(urls):
    with conn:
        conn.executemany(
            "INSERT OR IGNORE INTO frontier(url) VALUES (?)",
            [(u,) for u in urls]
        )

def next_url():
    row = conn.execute(
        "SELECT url FROM frontier WHERE visited = 0 ORDER BY rowid LIMIT 1"
    ).fetchone()
    return row[0] if row else None

def mark_visited(url):
    with conn:
        conn.execute("UPDATE frontier SET visited = 1 WHERE url = ?", (url,))

enqueue([BASE])
base_domain = urlparse(BASE).netloc
count = 0

while count < MAX_PAGES:
    url = next_url()
    if not url:
        break
    try:
        resp = session.get(url, timeout=10)
    except requests.RequestException:
        mark_visited(url)
        continue
    mark_visited(url)
    if resp.status_code != 200:
        continue
    count += 1
    soup = BeautifulSoup(resp.text, "html.parser")
    links = []
    for a in soup.find_all("a", href=True):
        link = urljoin(url, a["href"]).split("#")[0]
        if urlparse(link).netloc == base_domain:
            links.append(link)
    enqueue(links)

print(f"Crawled {count} pages")
conn.close()

INSERT OR IGNORE on the primary key handles dedup at the database level, so a URL added twice is silently discarded. ORDER BY rowid LIMIT 1 pulls the next unvisited URL in insertion order, which gives FIFO (BFS) behavior. The database file survives crashes with every URL’s visited flag intact, and restarting the script picks up from the first row where visited = 0.

The write cost is 1-2 ms per URL on a local SQLite file with WAL mode enabled. For crawls where fetches take 50-500 ms, that overhead is invisible. If it isn’t, add conn.execute("PRAGMA journal_mode=WAL") right after opening the connection to switch to write-ahead logging.

Beyond SQLite

Two more tools show up when a crawl genuinely outgrows a single SQLite file. A bloom filter (via pybloom-live or similar) gives probabilistic dedup at around 1.5 MB per million URLs, useful when the exact-answer index in SQLite starts costing multi-GB past 10 million URLs. Redis works as a shared frontier when the crawler needs to run across multiple processes or machines. Both are worth knowing about, but neither is needed until the crawl is well past what SQLite handles comfortably.

For crawls under 100,000 URLs, the in-memory pattern works fine. Between 100,000 and 10 million, the SQLite frontier above handles both queue and dedup on a single machine.

Scaling up with Scrapy

Scrapy packages retries, rate limiting, deduplication, output formats, and middleware as configuration. Most production features come from adjusting settings rather than writing new code, so switching to it usually means writing less overall.

Here’s a complete Scrapy spider for books.toscrape.com with the production settings that matter.

"""Scrapy spider for books.toscrape.com with production settings."""
from scrapy.crawler import CrawlerProcess
from scrapy.spiders import Spider


class BooksSpider(Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/"]

    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "USER_AGENT": "sample-python-crawler/1.0 (contact@example.com)",
        "DOWNLOAD_DELAY": 0.3,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 5,
        "AUTOTHROTTLE_ENABLED": True,
        "RETRY_HTTP_CODES": [429, 500, 502, 503, 504],
        "RETRY_TIMES": 3,
        "CLOSESPIDER_PAGECOUNT": 50,
        "FEEDS": {"books.json": {"format": "json", "overwrite": True}},
        "LOG_LEVEL": "WARNING",
    }

    def parse(self, response):
        # Book detail page? Yield the item.
        title = response.css("div.product_main h1::text").get()
        if title:
            yield {
                "url": response.url,
                "title": title,
                "price": response.css("p.price_color::text").get(),
            }

        # Follow all same-domain links.
        for href in response.css("a::attr(href)").getall():
            yield response.follow(href, callback=self.parse)


if __name__ == "__main__":
    process = CrawlerProcess()
    process.crawl(BooksSpider)
    process.start()

Every setting above replaces code the basic crawler wrote by hand. ROBOTSTXT_OBEY = True replaces the RobotFileParser and .can_fetch() calls. AUTOTHROTTLE_ENABLED = True replaces the fixed time.sleep(DELAY) with an adaptive delay based on server response times. RETRY_HTTP_CODES and RETRY_TIMES replace the manual retry loop with backoff. FEEDS exports the yielded items to JSON without extra code. CLOSESPIDER_PAGECOUNT is Scrapy’s built-in page cap.

Pipelines are Scrapy’s middleware that runs on each yielded item. Common uses include validation, deduplication by content field (not URL), database storage, and image downloading. When a simple FEEDS export isn’t enough, add a pipeline class in pipelines.py and enable it in the ITEM_PIPELINES setting.

For sites that publish a sitemap.xml, Scrapy’s SitemapSpider reads the sitemap and enqueues all URLs directly instead of crawling by following links. It finds pages that aren’t reachable from the homepage and skips the discovery phase. Swap Spider for SitemapSpider and set sitemap_urls = ["https://target.com/sitemap.xml"], then the parse callback stays the same.

Crawling JavaScript-heavy sites with Playwright

Some sites don’t put content in HTML. The data lives in JavaScript that runs after the page loads. Requests-based crawlers hit an empty shell where the actual content should be. Playwright fixes this by launching a real browser, running the JavaScript, waiting for the DOM to fill in, and returning the rendered HTML.

To test whether a target needs headless, fetch it with requests and look for the content you expect. If it’s missing, the site needs JavaScript execution.

"""Minimal Playwright crawler for JS-rendered sites."""
import asyncio
from urllib.parse import urlparse
from playwright.async_api import async_playwright


BASE = "https://quotes.toscrape.com/js/"
MAX_PAGES = 20


async def crawl():
    visited = set()
    queue = [BASE]
    base_domain = urlparse(BASE).netloc

    async with async_playwright() as p:
        browser = await p.chromium.launch()
        context = await browser.new_context()
        page = await context.new_page()

        while queue and len(visited) < MAX_PAGES:
            url = queue.pop(0)
            if url in visited:
                continue
            try:
                await page.goto(url, wait_until="networkidle", timeout=15000)
            except Exception:
                continue
            visited.add(url)
            quotes = await page.eval_on_selector_all(
                ".quote .text",
                "elements => elements.map(el => el.textContent)",
            )
            if quotes:
                print(f"{url}: {len(quotes)} quotes")
            links = await page.eval_on_selector_all(
                "a[href]",
                "elements => elements.map(el => el.href)",
            )
            for link in links:
                clean = link.split("#")[0]
                if (urlparse(clean).netloc == base_domain
                        and clean not in visited
                        and clean not in queue):
                    queue.append(clean)

        await browser.close()

    return len(visited)


if __name__ == "__main__":
    pages = asyncio.run(crawl())
    print(f"Crawled {pages} pages")

wait_until="networkidle" waits for the browser’s network to go quiet for 500ms, which usually means the initial JS fetch cycle is done. Some sites keep fetching (polling, analytics) and never go idle. Use wait_until="domcontentloaded" with a manual page.wait_for_selector(".quote") in that case.

Reusing a single page and context across URLs is the biggest performance improvement over the naive pattern. Launching a fresh browser per URL adds 500-1500 ms of overhead each time. The pattern above amortizes the launch cost across all pages.

On the same URL, Playwright runs about 2.2x slower than a requests call. That’s the cost of launching Chromium, running the JS, and waiting for the network to idle. It only pays off when the content really lives in JavaScript. On a static page, requests returns the same content in half the time. Test each target before defaulting to headless.

Speed and scaling, benchmarked

I ran the four Python crawler approaches against books.toscrape.com. Each crawled to a 100-page budget, with async tools capped at concurrency=5. Numbers are medians of 3 runs per tool on Python 3.13.7.

On chart:

Scrapy tops throughput at 14.8 pages/sec, roughly 4x sync Requests. httpx.AsyncClient closes 80% of that gap at 11.6 pages/sec with a fraction of Scrapy’s setup, running as an async loop and gather without a framework or project layout. Both async approaches beat sync Requests decisively because HTTP wait time dominates on a fast target, and concurrency turns that wait into overlap.

Crawlee for Python’s result stands out. At 3.9 pages/sec, it lands roughly where sync Requests is, despite being async, using the same concurrency limit, and being explicitly designed for scale. The reason is its default HTTP client (impit) performs full browser TLS fingerprinting on every request. That fingerprinting is what makes Crawlee useful against sites that reject default HTTP clients. On a target with no defenses (books.toscrape.com), the fingerprinting overhead is pure cost. Swap impit for HttpxCrawler and Crawlee matches httpx throughput.

Memory scales with framework surface. Requests+BS4 uses the least at 55 MB. httpx adds about 15 MB for the async runtime. Scrapy adds another 30 MB for the framework. Crawlee tops the list at 100 MB, driven by fingerprinting state and asyncio task tracking.

Concurrency=5 is a conservative default for a public target. For a site that can handle load, raising to 10 or 20 pushes async throughput closer to Scrapy’s ceiling. Past 20, the limits shift from HTTP wait time to either the target’s rate limits (429 responses, connection resets) or the local machine’s CPU on HTML parsing. Rate limits show up first on almost every real target. Start at 5, monitor for 429s, and raise carefully.

Common errors and production concerns

Crawls that grow past a few hundred pages hit failure modes the 20-page demo never sees. Transient errors show up, rate limits kick in, and target servers start noticing traffic patterns. Handling each requires different code, not a bigger crawler.

Retriable vs permanent errors

Not every failed request should be retried. Retriable errors are things that might succeed on a second attempt. Permanent errors won’t.

Exponential backoff with jitter is the standard retry pattern. Base delay doubles each attempt, plus a random 0-500ms to avoid the thundering herd when multiple workers all hit the same 502 at once. Cap at 3-5 attempts total. Scrapy and Crawlee both do this automatically with RETRY_HTTP_CODES and max_request_retries. Custom crawlers wrap the fetch call in a small retry helper.

Rate limiting patterns

Rate limiting looks different in each tool. Scrapy uses AUTOTHROTTLE_ENABLED = True combined with DOWNLOAD_DELAY, both configured in the spider’s custom_settings. httpx.AsyncClient needs an asyncio.Semaphore(N) around each fetch call. Custom Requests+BS4 uses time.sleep(delay) after each successful fetch. Crawlee configures rate through ConcurrencySettings(desired_concurrency=N).

Resumability is the other production concern. The SQLite frontier from earlier lets a crashed crawler resume by querying for URLs where visited = 0. In-memory crawlers lose their state on crash. Any crawl expected to take longer than a few minutes should persist frontier state to disk.

Anti-bot mitigation

Sites that notice crawlers block them through User-Agent checks (rejecting python-requests), rate limits (429 or IP blocks), or TLS fingerprint checks (rejecting Python’s default cipher order). Handle each in escalating order.

Rotating User-Agents helps if the target blocks based on the default python-requests header. Real browsers keep the same UA within a session, so request-level rotation flags the crawler as a bot. A pool of realistic UAs, one per worker session, is the pattern that works.

import random
import requests

UA_POOL = [
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
]

def make_session():
    session = requests.Session()
    session.headers["User-Agent"] = random.choice(UA_POOL)
    return session

# One session per worker, UA fixed for that worker's lifetime
session = make_session()
resp = session.get("https://books.toscrape.com/")

Proxy rotation is the next step when IPs get blocked. requests.Session supports proxies through the proxies argument. Rotate at the session level, same as the User-Agent.

PROXY_POOL = [
    "http://user:pass@proxy1.example.com:8080",
    "http://user:pass@proxy2.example.com:8080",
    "http://user:pass@proxy3.example.com:8080",
]

def make_session():
    session = requests.Session()
    session.headers["User-Agent"] = random.choice(UA_POOL)
    proxy = random.choice(PROXY_POOL)
    session.proxies = {"http": proxy, "https": proxy}
    return session

When defenses escalate to JA3 fingerprinting or Cloudflare/Datadome tokens, the DIY approach hits its limits. Crawlee for Python’s impit handles TLS fingerprinting at a real throughput cost. For heavier lifting, managed scraping APIs like HasData’s Web Scraping API handle proxy rotation and anti-bot bypass as a service, which trades per-request cost for less crawler maintenance.

FAQ

What’s the difference between crawling and scraping?

A crawler discovers URLs by following links. A scraper extracts data from a page you already have. Most real projects need both, with the crawler walking the site and the scraper pulling fields from each page.

Do I need Scrapy or can I use Requests + BeautifulSoup?

Requests + BeautifulSoup works for crawls under a few hundred pages. Past that, Scrapy’s built-in retries, throttling, and pipelines save more code than they cost to learn. The threshold is fuzzy but shows up somewhere between 500 and a few thousand pages in practice.

Can Python crawl JavaScript-heavy sites?

Yes, through Playwright or Selenium. Both launch a real browser, run the JavaScript, and return the rendered DOM. The cost is about 2x slower than a requests call on the same URL, measured on a JS-rendered target. For static content, requests returns the same data faster.

What’s the fastest Python crawler?

Scrapy on a static target, at 14.8 pages/sec in my benchmark (4x sync Requests, 1.3x httpx.AsyncClient with concurrency=5). Playwright is faster than nothing on JS-rendered targets but slower than any HTTP client on static pages. Framework overhead only pays off past a few hundred pages.

How do I avoid getting blocked?

Respect rate limits (start at 5 concurrent requests, monitor for 429), set a realistic User-Agent, and rotate residential IPs if the target blocks by IP. Escalate in this order, starting with rate limits and moving through UA rotation, proxies, and then managed anti-bot services. Aggressive crawling gets aggressive counter-measures.

Conclusion

For static targets up to a few hundred pages, the 40-line Requests + BeautifulSoup crawler with robots.txt, rate limit, and retry is enough. Between a few hundred and a few thousand pages on a static site, Scrapy’s built-in machinery saves more code than it costs. Past that, the choice between async httpx, Scrapy, or Crawlee depends on whether the target blocks default HTTP clients. Crawlee’s fingerprinting is worth its 4x throughput cost only when defenses are active.

For JavaScript-heavy targets, Playwright fits when the data lives in the rendered DOM. The 2x latency cost is a real trade-off, and testing each target with requests first tells whether headless is required or wasted. When defenses push into JA3 fingerprinting and IP-based blocking, the DIY stack starts costing more engineering time than it saves. A managed scraping API is worth pricing out at that point, since the maintenance burden of a proxy pool, fingerprinting, and retry logic adds up fast.