11 Python Web Scraping Libraries, Benchmarked (2026)

There is no single best Python library for web scraping, and the benchmark below shows why. We implemented the same task, collecting 200 rows from a paginated listing, in ten stacks and measured each one. The spread runs from 3 seconds and 43 MB for the fastest HTTP stack to 32 seconds for the slowest browser stack and 1.4 GB for the hungriest, and exactly one of the ten got past the anti-bot probe, SeleniumBase’s UC mode. This guide groups the libraries and tools the way the choice actually works, HTTP clients, parsers, browser automation, and frameworks, with a runnable example and the measured cost of each.

Choosing by Task

The short version fits in one lookup table:

Every claim about speed or weight below comes from the same measured run.

The Benchmark

Each stack ran the same task, collecting 200 hockey-team rows from a paginated sandbox listing (scrapethissite.com, 8 sequential pages), 10 times in its own fresh virtual environment on the same machine, medians reported. Memory is the peak RSS of the whole process tree, dependencies are what pip install actually brings into an empty venv, and the anti-bot column is what one fetch of stackoverflow.com/questions, sent with a Chrome User-Agent, returns through that stack. The 11 libraries combine into these ten stacks. urllib3 is measured inside every requests stack, Scrapy appears twice (alone and paired with Playwright through the scrapy-playwright plugin), Selenium and Playwright each run once more under a stealth layer (the SeleniumBase framework’s UC mode and the playwright-stealth plugin), and the unmaintained Pyppeteer stays out of the harness. The run used requests 2.34.2, httpx 0.28.1, curl_cffi 0.16.3, BeautifulSoup 4.15.0, lxml 6.1.3, parsel 1.11.0, Scrapy 2.18.0, Selenium 4.48.0, Playwright 1.62.0, scrapy-playwright 0.0.48, SeleniumBase 4.53.7, and playwright-stealth 2.0.3 on Python 3.14.

The HTTP stacks finished within 45 MB of memory and a 0.7-second band of each other, with requests + lxml the fastest at 3.01 s and BeautifulSoup 23% slower on identical network work, the price of its forgiving API. Scrapy’s framework overhead on a task this small is modest (3.46 s, 81 MB), and its 35 packages are the real weight. The plain browsers cost five to seven times the wall clock and twelve to seventeen times the memory of the lxml stack, with Playwright launching faster than Selenium (0.28 s against 1.57 s) and finishing the task faster too. playwright-stealth added a second of wall clock and 20 MB to plain Playwright, and SeleniumBase’s UC mode, which drives the installed Chrome in a visible window, posted the slowest task time of the ten at 32.06 s. The scrapy-playwright row shows what a scheduler does to browser cost: Scrapy runs the rendered fetches concurrently, so the same eight pages finish in 8.08 s, and the bill moves to memory, 1,460 MB of parallel browser pages, the heaviest row in the table.

Stack Overflow answered 403 to eight of the ten stacks, headless browsers included, and served Selenium its challenge page. playwright-stealth got the same 403 as plain Playwright. The one 200 came from SeleniumBase’s UC mode and its patched chromedriver, at the slowest task time of the ten. A single probe against a single target is a snapshot rather than a promise, and at this level of defense the library choice decides less than the infrastructure behind the request.

The absolute numbers belong to one machine and one target, so read the ratios, not the milliseconds. Each implementation is 21-36 lines against a public sandbox, so the whole run is small enough to replay in an evening.

HTTP Clients

The fetch layer decides what your scraper can reach before a single selector runs, and four clients cover the realistic range.

Requests

requests is the default HTTP client of the Python world and the shortest path from a URL to HTML. Sessions reuse connections across calls, headers and cookies are plain dictionaries, and nearly every scraping tutorial assumes it, so any error you hit already has an answer written up.

import requests

r = requests.get("https://www.scrapethissite.com/pages/simple/", timeout=20)
r.raise_for_status()
print(r.status_code, len(r.text), r.headers["content-type"])

It fetches, and that is the whole job description. No JavaScript, no HTTP/2, and its stock TLS fingerprint is one of the signals anti-bot vendors key on. For server-rendered targets without protection, none of that matters and requests wins on ubiquity.

HTTPX

httpx covers the same ground as requests with a modern core. The same API shape works sync and async, and HTTP/2 is one extra install away. If a project already runs on asyncio, httpx slots in where requests cannot.

import httpx

with httpx.Client(timeout=20) as client:
    r = client.get("https://www.scrapethissite.com/pages/simple/")
    r.raise_for_status()
    print(r.status_code, r.http_version, len(r.text))

On the benchmark it behaves like requests with a slightly heavier install (12 packages against 8). The async capability is the actual reason to pick it, since sequential scraping gains nothing from the switch. Three pages fetched concurrently look like this:

import asyncio

import httpx

async def main():
    async with httpx.AsyncClient(timeout=20) as client:
        pages = await asyncio.gather(
            *[client.get("https://www.scrapethissite.com/pages/simple/",
                         params={"page_num": n}) for n in (1, 2, 3)]
        )
    print([r.status_code for r in pages])

asyncio.run(main())

aiohttp does the same job for async-only codebases, and httpx covering both styles is what makes it the simpler default.

curl_cffi

curl_cffi wraps libcurl with browser impersonation, so the TLS and HTTP/2 fingerprints match a real Chrome. That helps exactly where the block is fingerprint-level, and our probe shows the limit of it. Stack Overflow still answered 403 to the impersonated client, the same as to requests, because its defense checks more than the handshake.

from curl_cffi import requests

r = requests.get("https://stackoverflow.com/questions", impersonate="chrome", timeout=20)
print(r.status_code, len(r.text))

The API mirrors requests closely enough that migration is mostly the import line, and the install is the lightest of the HTTP stacks (5 packages against 8 for requests and 12 for httpx). Where a plain client fails on the handshake alone, impersonation gets the page. Where the defense also reads behavior, IP reputation, or runs a JavaScript challenge, it does not.

urllib3

urllib3 is the connection-pool layer requests is built on, not a scraping library in its own right. Use it directly when you need pool-level control, retry objects, or zero extra dependencies, and use requests for everything else:

import urllib3

http = urllib3.PoolManager()
r = http.request("GET", "https://www.scrapethissite.com/pages/simple/", timeout=20.0)
print(r.status, len(r.data))

That control matters for connection tuning rather than scraping, so this guide treats it as context.

Parsers

Parsing is where the HTTP stacks actually differed in the benchmark, since their network work was identical.

BeautifulSoup4

Broken markup is the case for BeautifulSoup4. It accepts what stricter parsers reject, and its answers to malformed HTML are sensible defaults rather than exceptions, behind an API that reads like English.

import requests
from bs4 import BeautifulSoup

html = requests.get("https://www.scrapethissite.com/pages/simple/", timeout=20).text
soup = BeautifulSoup(html, "html.parser")
countries = soup.select("div.country")
first = countries[0]
print(len(countries), first.select_one("h3.country-name").get_text(strip=True))

The price is speed. The same task ran 3.01 s on lxml against 3.71 s on BeautifulSoup with the stock parser, and the gap is almost entirely parsing time, since the network half is identical. BeautifulSoup can hand the parsing to lxml as its backend (BeautifulSoup(html, "lxml")) while keeping the API.

lxml

lxml binds the C libraries libxml2 and libxslt, and it is the speed baseline the others chase. It speaks XPath natively, which matters once selections get conditional:

import requests
from lxml import html

page = requests.get("https://www.scrapethissite.com/pages/simple/", timeout=20).text
tree = html.fromstring(page)
names = tree.xpath("//h3[@class='country-name']/text()[normalize-space()]")
capitals = tree.xpath("//span[@class='country-capital']/text()")
print(len(capitals), names[0].strip(), capitals[0])

The fastest measured stack in the table pairs it with requests at 43 MB peak, and its XPath dialect is the one whose text-matching forms we measured on live pages.

parsel

parsel is Scrapy’s selector engine as a standalone package, and it ends the CSS-or-XPath argument by chaining both against the same tree:

import httpx
from parsel import Selector

html = httpx.get("https://www.scrapethissite.com/pages/simple/", timeout=20).text
sel = Selector(text=html)
rows = [
    {
        "name": c.css("h3.country-name::text").getall()[-1].strip(),
        "capital": c.css("span.country-capital::text").get(),
        "population": c.xpath(".//span[@class='country-population']/text()").get(),
    }
    for c in sel.css("div.country")
]
print(len(rows), rows[0])

It runs on lxml underneath, so the speed story is lxml’s, and anyone who later moves the code into a Scrapy spider keeps the selectors unchanged.

Browser Automation

When the page renders its data dynamically, after JavaScript runs, the fetch layer becomes a browser, and the benchmark table above prices that switch.

Playwright

Playwright is the current default answer for JavaScript-rendered pages. One pip install playwright plus playwright install chromium brings a managed browser, auto-waiting locators, and an API that needs no manual WebDriverWait choreography:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://www.scrapethissite.com/pages/ajax-javascript/")
    page.click("a[id='2015']")
    page.wait_for_selector("td.film-title")
    titles = page.locator("td.film-title").all_inner_texts()
    browser.close()
print(len(titles), titles[0])

The target here loads its data over XHR, so the HTTP stacks above see an empty shell and the browser sees the table. The measured cost of that power is the browser itself, 14.79 s for the task that lxml finished in 3.01 s, at 516 MB against 43. Spend it on pages that need it. The playwright-stealth plugin patches the JavaScript properties headless Chromium leaks, navigator.webdriver first among them, with one call per page over the code above. On our probe it bought nothing, the same 403 as plain Playwright for roughly a second of extra wall clock.

Selenium

Selenium automates a real browser for the same JavaScript-rendered pages, with a longer history and the widest set of integrations. Since 4.6 its Selenium Manager downloads drivers automatically, so setup is one install:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
driver.get("https://www.scrapethissite.com/pages/ajax-javascript/")
driver.find_element(By.ID, "2015").click()
WebDriverWait(driver, 15).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, "td.film-title"))
)
titles = [el.text for el in driver.find_elements(By.CSS_SELECTOR, "td.film-title")]
driver.quit()
print(len(titles), titles[0])

Selenium 4 removed the old find_element_by_id family of calls, and the block above is the current API. The explicit waits Playwright hides are visible here, which some teams prefer for debuggability, and each Selenium configuration carries a measured cost. SeleniumBase wraps that driver management in a framework of its own, and its UC mode swaps in a patched chromedriver and drives the installed Chrome in a visible window. That mode returned the table’s only 200 on the anti-bot probe, and the cost is the slowest task time of the ten, 32.06 s at 894 MB, with 61 packages in a fresh venv.

Pyppeteer, and Why Not to Start with It

Pyppeteer was an unofficial Python port of Puppeteer, and it is unmaintained. The project itself recommends migrating, its bundled Chromium is years old, and every job it did is covered by Playwright above. It stays in this list only because old tutorials still recommend it, and new projects start with Playwright.

Frameworks

Past a few hundred pages, scheduling and retries matter more than parsing speed, and that is the layer frameworks own.

Scrapy

Scrapy is a crawling framework rather than a library, and the difference shows exactly at scale. Scheduling, concurrent requests, retries, throttling, and export pipelines are configuration, not code you write:

import scrapy
from scrapy.crawler import CrawlerProcess

results = []

class CountriesSpider(scrapy.Spider):
    name = "countries"
    custom_settings = {"LOG_ENABLED": False, "ROBOTSTXT_OBEY": True}
    start_urls = ["https://www.scrapethissite.com/pages/simple/"]

    def parse(self, response):
        for c in response.css("div.country"):
            # in a real spider, yield the dict instead of appending
            results.append({
                "name": c.css("h3.country-name::text").getall()[-1].strip(),
                "capital": c.css("span.country-capital::text").get(),
            })

process = CrawlerProcess()
process.crawl(CountriesSpider)
process.start()
print(len(results), results[0])

The benchmark shows the framework tax and the framework payoff in one row, 3.46 s and 81 MB on a task this small with 35 packages installed, while the same spider, built out the way the Scrapy tutorial shows, absorbs a crawl orders of magnitude larger by changing settings. Throttling is configuration too, AUTOTHROTTLE_ENABLED backs off when a target starts answering 429.

When the pages need JavaScript, Scrapy does not switch tools, it swaps the download handler. The scrapy-playwright plugin routes requests through a Playwright browser while the scheduler, retries, and pipelines stay Scrapy’s:

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy_playwright.page import PageMethod

results = []

class FilmsSpider(scrapy.Spider):
    name = "films"
    custom_settings = {
        "LOG_ENABLED": False,
        "ROBOTSTXT_OBEY": True,
        # route downloads through Playwright, keep everything else Scrapy
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
    }

    async def start(self):
        yield scrapy.Request(
            "https://www.scrapethissite.com/pages/ajax-javascript/",
            meta={"playwright": True, "playwright_page_methods": [
                PageMethod("click", "a[id='2015']"),
                PageMethod("wait_for_selector", "td.film-title"),
            ]},
        )

    def parse(self, response):
        results.extend(t.strip() for t in response.css("td.film-title::text").getall())

process = CrawlerProcess()
process.crawl(FilmsSpider)
process.start()
print(len(results), results[0] if results else None)

The run prints the same 16 Spotlight as the plain browser examples, and the benchmark row above is this pairing on the 200-row task: 8.08 s, because the scheduler renders pages concurrently, at 1,460 MB of parallel browser memory. Two honest caveats from running it. On Scrapy 2.18 the seeds must come from async def start(), since a start_requests override never gets called with the asyncio reactor and the spider closes silently with zero rows. And on Windows the plugin’s loop thread races the shutdown, which prints a harmless Event loop is closed trace after the data on most runs, so treat the pairing as Linux-first. For one page it is overkill, and against BeautifulSoup the decision is a measured comparison of its own.

When a Library Is Not Enough

A blocked target is a different problem from a slow parser. A web scraping API is the outsourcing option for it. It takes the URL, runs the proxies and rendering on the service side, and returns HTML or extracted JSON, so the library choice above collapses back into one HTTP call with requests. It is not a library and has no benchmark row. You buy it when maintaining proxies and headless browsers costs more than the data is worth.

Conclusion

The groups make the choice short. For a server-rendered target, take requests (or httpx on an async codebase) with lxml when speed matters and BeautifulSoup when the markup does not inspire trust. When the first plain fetch comes back 403, try curl_cffi before reaching for a browser. Pages that render their data with JavaScript go to Playwright first, to Selenium where your team’s history argues for it, and never to Pyppeteer. Recurring structured crawls belong to Scrapy, 35 packages and all, because the alternative is writing your own scheduler. The benchmark table above is the same advice in numbers.