Web Scraping with Scrapy in 2026 (Benchmarked)

The scrapy startproject template includes CONCURRENT_REQUESTS_PER_DOMAIN = 16 and AUTOTHROTTLE_ENABLED = True as commented suggestions. Tutorials copy those values and recommend CSS for simple pages, XPath for complex ones. Nobody publishes the numbers behind those choices, so I ran the benchmarks on books.toscrape.com with Scrapy 2.17.0 and Python 3.13.

Here’s what the runs showed.

You need Python 3.11+ and a virtualenv. Async experience is optional. Twisted’s event loop stays hidden until you write custom middleware.

What is Scrapy and when to use it

Scrapy is a full Python web scraping framework built on Twisted, the async networking library. You get a spider class, a scheduling engine, HTTP fetching with cookies and retries, pluggable middleware, output pipelines, and CLI tooling from one install:

pip install scrapy

Scrapy’s engine sits in the middle. Your spider yields Request objects to a scheduler queue, which releases them subject to CONCURRENT_REQUESTS and CONCURRENT_REQUESTS_PER_DOMAIN. Downloader middleware (proxy, User-Agent, retry) runs before each fetch, the downloader hits the network, and response middleware runs on the way back. Whatever your callback yields (items or fresh requests) returns to the engine. Items pass through pipelines for validation, cleaning, and storage.

Use Scrapy when you’re crawling many pages, need concurrent HTTP without writing your own event loop, and want the split between fetching, parsing, and storage. For a single page with BeautifulSoup, Scrapy is overkill. The Scrapy vs BeautifulSoup benchmark covers that comparison in depth. Pages that render fully in JavaScript need scrapy-playwright.

Installing Scrapy and creating a project

Set up a virtualenv, install Scrapy, and verify the version:

python -m venv .venv
source .venv/bin/activate  # macOS/Linux
# or: .venv\Scripts\activate  # Windows
pip install -U scrapy
scrapy version

The last command should print Scrapy 2.17.0 or newer.

Create a new project:

scrapy startproject books
cd books

This generates:

books/
├── scrapy.cfg              # deployment config
└── books/
    ├── __init__.py
    ├── items.py            # structured data classes
    ├── middlewares.py      # spider/downloader middleware
    ├── pipelines.py        # item pipelines (validation, storage)
    ├── settings.py         # project-wide config (concurrency, delays, UA)
    └── spiders/            # your spider modules go here
        └── __init__.py

spiders/ is where spider modules live. settings.py centralizes concurrency, delays, User-Agent, and pipeline registration. pipelines.py holds item processors for validation, cleaning, and storage. items.py defines typed field classes when you want them. middlewares.py handles custom request or response processing and stays empty on most crawls.

Building your first spider

Generate a spider file:

scrapy genspider books books.toscrape.com

That creates books/spiders/books.py. Replace its contents with:

# books/spiders/books.py
from scrapy import Spider


class BooksSpider(Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/"]

    def parse(self, response):
        for article in response.css("article.product_pod"):
            yield {
                "title": article.css("h3 a::attr(title)").get(),
                "price": article.css("p.price_color::text").get(),
                "rating": article.css("p.star-rating::attr(class)").get().split()[-1],
            }

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

Read the file top to bottom. start_urls is the seed list Scrapy calls parse on. Inside parse, response.css("article.product_pod") returns a SelectorList with one selector per book. Each yield {...} sends a dict to the pipelines. The last block reads href from the next pagination link and calls response.follow, which resolves the relative URL against the current page and queues a new request. Scrapy keeps calling parse until pagination runs out.

Run it:

scrapy crawl books -o books.jsonl

The -o flag exports items to a file. .jsonl (JSON Lines) is the safest format. Items stream out one per line as they arrive, so a crash mid-crawl leaves you with valid partial output. .json and .csv also work but only write the full document when the crawl ends.

For interactive testing, use scrapy shell:

scrapy shell "https://books.toscrape.com/"

You land in a REPL with response pre-loaded. Try selectors against the live page before committing them to the spider:

>>> response.css("article.product_pod h3 a::attr(title)").getall()[:3]
['A Light in the Attic', 'Tipping the Velvet', 'Soumission']

>>> response.css("li.next a::attr(href)").get()
'catalogue/page-2.html'

Every selector that works in the shell drops directly into the spider.

For sites where every internal link should be crawled (not just specific pagination), Scrapy provides CrawlSpider with LinkExtractor rules. Instead of manually calling response.follow, define URL patterns to follow:

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor


class BooksCrawlSpider(CrawlSpider):
    name = "books-crawl"
    start_urls = ["https://books.toscrape.com/"]

    rules = (
        Rule(LinkExtractor(allow=r"catalogue/"), callback="parse_book", follow=True),
    )

    def parse_book(self, response):
        yield {
            "title": response.css("div.product_main h1::text").get(),
            "price": response.css("p.price_color::text").get(),
        }

CrawlSpider follows every link matching the allow regex and calls parse_book on each page. Use it when the target site has a consistent URL pattern for the content you want. For sites with pagination plus individual page follows, the manual response.follow pattern stays clearer.

CSS vs XPath selectors, benchmarked

Scrapy supports both CSS selectors and XPath expressions. The same three fields, extracted two ways:

# CSS
for article in response.css("article.product_pod"):
    yield {
        "title": article.css("h3 a::attr(title)").get(),
        "price": article.css("p.price_color::text").get(),
        "rating": article.css("p.star-rating::attr(class)").get().split()[-1],
    }

# XPath
for article in response.xpath("//article[contains(@class, 'product_pod')]"):
    yield {
        "title": article.xpath(".//h3/a/@title").get(),
        "price": article.xpath(".//p[contains(@class, 'price_color')]/text()").get(),
        "rating": article.xpath(".//p[contains(@class, 'star-rating')]/@class").get().split()[-1],
    }

CSS syntax is shorter for class-based lookups. XPath wins for anything involving ancestor traversal, following-sibling axes, or text() predicates. //p[contains(text(), 'In stock')] has no CSS equivalent short of :has(), and :has() support in the underlying cssselect stack is patchy.

Tutorials often say CSS is faster because its syntax is simpler. The measurement disagrees.

Benchmark setup

• Target: books.toscrape.com catalogue page (20 books)
• Fields per book: title, price, rating class
• 5000 iterations each style, 500-iteration warmup
• Scrapy 2.17.0, Python 3.13, single Selector instance per iteration

Results

XPath is about 10% faster on this page. The reason lives inside Scrapy itself. CSS selectors compile to XPath via the cssselect library on every evaluation, adding a translation step that XPath queries skip. Outside Scrapy, XPath still wins in lxml by a similar margin.

The gap is small in absolute terms. On a 100-page crawl at this density, XPath saves roughly 40 milliseconds total, well below network noise. It shows up on sites with hundreds of items per page or in scheduled runs where cumulative cost adds up.

Which selector to pick by situation:

Items, ItemLoaders, and pipelines

Between the spider callback and final storage, Scrapy stages data through Items, ItemLoaders, Pipelines, and Feed exports. Each covers a specific step.

Items

Items are typed data classes. You define fields once, get a dict-like interface with schema validation. Beats raw dicts when the crawl grows past 3-5 fields per item.

# books/items.py
from scrapy import Field, Item


class BookItem(Item):
    title = Field()
    price = Field()
    rating = Field()
    availability = Field()

In the spider, replace the dict yield with an Item instance:

from books.items import BookItem

def parse(self, response):
    for article in response.css("article.product_pod"):
        item = BookItem()
        item["title"] = article.css("h3 a::attr(title)").get()
        item["price"] = article.css("p.price_color::text").get()
        item["rating"] = article.css("p.star-rating::attr(class)").get().split()[-1]
        yield item

Accessing an unknown field raises KeyError at write time. Typos in field names fail fast instead of appearing as missing columns in output.

Cleaning with ItemLoaders

ItemLoaders wrap Item assignment with input and output processors. Instead of stripping and casting each field inline in the spider, register the transformations once and let the loader apply them at yield time.

# books/items.py
from itemloaders.processors import MapCompose, TakeFirst
from scrapy import Field, Item


def parse_price(value: str) -> float:
    # extend the replace() chain for €, ¥, or other currencies
    return float(value.replace("£", "").replace("$", "").strip())


class BookItem(Item):
    title = Field(
        input_processor=MapCompose(str.strip),
        output_processor=TakeFirst(),
    )
    price = Field(
        input_processor=MapCompose(parse_price),
        output_processor=TakeFirst(),
    )
    rating = Field(output_processor=TakeFirst())

Then in the spider:

from scrapy.loader import ItemLoader
from books.items import BookItem

def parse(self, response):
    for article in response.css("article.product_pod"):
        loader = ItemLoader(item=BookItem(), selector=article)
        loader.add_css("title", "h3 a::attr(title)")
        loader.add_css("price", "p.price_color::text")
        loader.add_css("rating", "p.star-rating::attr(class)")
        yield loader.load_item()

MapCompose(str.strip) runs str.strip on every input value. TakeFirst() picks the first non-empty result. Cleaning moves out of the parse method and into the Item definition.

Pipelines

Pipelines run after items leave the spider. Register them in settings.py with a numeric priority (lower runs first). Typical use cases are validation, which drops items missing required fields, and storage, which writes to CSV, JSON, SQLite, PostgreSQL, MongoDB, S3, or anything else with a Python client.

A validation pipeline that drops books priced below £5 (likely parse errors):

# books/pipelines.py
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem


class PriceValidationPipeline:
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        price = adapter.get("price")
        if price is None or price < 5.0:
            raise DropItem(f"Suspicious price for {adapter.get('title')!r}: {price}")
        return item

An SQLite storage pipeline:

# books/pipelines.py (continued)
import sqlite3
from itemadapter import ItemAdapter


class SQLiteStoragePipeline:
    def open_spider(self, spider):
        self.conn = sqlite3.connect("books.db")
        self.conn.execute(
            "CREATE TABLE IF NOT EXISTS books "
            "(title TEXT, price REAL, rating TEXT)"
        )

    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        self.conn.execute(
            "INSERT INTO books VALUES (?, ?, ?)",
            (adapter["title"], adapter["price"], adapter["rating"]),
        )
        self.conn.commit()
        return item

    def close_spider(self, spider):
        self.conn.close()

Enable both in settings.py:

ITEM_PIPELINES = {
    "books.pipelines.PriceValidationPipeline": 100,
    "books.pipelines.SQLiteStoragePipeline": 200,
}

Validation runs first (priority 100), storage second (200). If validation drops the item, storage never sees it.

Feed exports

For quick outputs without writing a pipeline, use FEEDS in settings.py:

# books/settings.py
FEEDS = {
    "output/%(name)s-%(time)s.jsonl": {"format": "jsonlines"},
    "output/%(name)s-%(time)s.csv": {"format": "csv"},
}

Every crawl writes both files, with the spider name and timestamp in the filename. %(name)s interpolates to books, %(time)s to an ISO timestamp. Feed exports live at a lower layer than pipelines. Items pass through pipelines first, then get serialized to feed files.

Passing -o filename.jsonl on the scrapy crawl command wires up a one-off feed export without touching settings.py.

Production settings and concurrency tuning

Production Scrapy tuning happens across politeness settings, concurrency, retries, and spider verification. Each has defaults you can ship with plus one or two knobs worth turning.

Settings that matter

Baseline settings.py after scrapy startproject, edited for a real crawl:

# books/settings.py
BOT_NAME = "books"
SPIDER_MODULES = ["books.spiders"]
NEWSPIDER_MODULE = "books.spiders"

# Politeness
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36"
ROBOTSTXT_OBEY = True

# Concurrency
CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 10  # sweet spot from the benchmark
DOWNLOAD_DELAY = 0.25

# Retries
RETRY_ENABLED = True
RETRY_HTTP_CODES = [500, 502, 503, 504, 429]
RETRY_TIMES = 3

USER_AGENT needs a real browser string. Scrapy’s default Scrapy/2.17.0 (+https://scrapy.org) gets flagged by most anti-bot layers. ROBOTSTXT_OBEY = True reads robots.txt and skips disallowed paths. DOWNLOAD_DELAY = 0.25 puts 250ms between requests to the same domain (randomized ±50% by Scrapy). Proxy rotation and header rotation for stubborn anti-bot sites live in the HasData Web Scraping API or a rotated proxy setup you manage yourself.

Concurrency and AutoThrottle, benchmarked

Tutorials pick CONCURRENT_REQUESTS_PER_DOMAIN without explaining why. Here’s the throughput at c=1, 5, 10, 16, and 32 on the same 50-page books.toscrape.com crawl, plus AutoThrottle for comparison. Median of 3 runs per configuration.

On the chart:

Throughput peaks at CONCURRENT_REQUESTS_PER_DOMAIN = 10. Below that, network idle time dominates. Above, the demo target starts throttling and the extra slots sit blocked waiting for responses.

AutoThrottle uses AUTOTHROTTLE_TARGET_CONCURRENCY to keep an adaptive number of concurrent requests based on response latency. When the site takes longer to respond, it backs off. Throughput drops about 36% compared to a tuned fixed setting. That’s worth it for unknown sites where the throttling threshold is unclear:

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 10.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 8.0

For sites you own or already know the limits of, disable AutoThrottle and set a fixed CONCURRENT_REQUESTS_PER_DOMAIN.

Retries and error handling

Any real crawl hits transient failures. Scrapy’s built-in retry middleware retries 500, 502, 503, 504, 522, 524, 408, 429 and network errors by default. Configure the list and count in settings.py:

RETRY_HTTP_CODES = [500, 502, 503, 504, 429]
RETRY_TIMES = 3

RETRY_TIMES counts on top of the initial request. A value of 3 means the request tries 4 times total before giving up. The built-in delay between retries is fixed. Exponential backoff and other retry patterns need custom middleware.

For non-standard failures (spider callback exceptions, custom bad responses), attach an errback:

import scrapy
from scrapy import Spider


class BooksSpider(Spider):
    name = "books"

    def start_requests(self):
        for url in ["https://books.toscrape.com/"]:
            yield scrapy.Request(url, callback=self.parse, errback=self.on_error)

    def on_error(self, failure):
        request = failure.request
        self.logger.warning(f"Failed {request.url}: {failure.value}")

Verifying spiders with contracts

Spider contracts are small assertions attached to parse callbacks. scrapy check runs them against the live URL, catching selector regressions before you push a broken spider:

def parse(self, response):
    """Extract books from a listing page.

    @url https://books.toscrape.com/
    @returns items 20 20
    @scrapes title price rating
    """
    for article in response.css("article.product_pod"):
        yield {
            "title": article.css("h3 a::attr(title)").get(),
            "price": article.css("p.price_color::text").get(),
            "rating": article.css("p.star-rating::attr(class)").get().split()[-1],
        }

Run:

scrapy check books

@url fetches the test page. @returns items 20 20 asserts the callback yields exactly 20 items. @scrapes title price rating asserts every item has those three fields non-empty. If books.toscrape.com changes its markup and the CSS selectors stop matching, scrapy check fails loudly before the next production crawl silently produces empty output.

Handling JavaScript-heavy sites

Scrapy fetches HTML and hands it to your parse callback. It doesn’t run JavaScript. Content that only appears after the browser executes scripts stays invisible to response.css and response.xpath.

The workaround is scrapy-playwright, a middleware that runs each request through a headless browser before your callback sees the response. (scrapy-splash is the older Splash-based alternative, but development stalled and Playwright is now the default.) Install it:

pip install scrapy-playwright
playwright install chromium

Enable it in settings.py:

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
PLAYWRIGHT_BROWSER_TYPE = "chromium"

Add playwright=True to any request that needs a browser:

import scrapy


class QuotesJsSpider(scrapy.Spider):
    name = "quotes-js"

    def start_requests(self):
        yield scrapy.Request(
            "https://quotes.toscrape.com/js/",
            meta={"playwright": True},
            callback=self.parse,
        )

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
            }

The rest of the spider stays the same. Selectors, pipelines, and settings keep working. Passing playwright_include_page=True exposes the raw Page object via response.meta for Playwright patterns like waits, network interception, and session reuse.

Playwright inside Scrapy handles a few thousand pages a day. Beyond that, browser overhead dominates. Each fetch spins up a Chromium tab that eats CPU and memory. For large-scale JavaScript rendering, offload to a browser farm or the HasData Web Scraping API, which handles rendering and rotation server-side.

Deploying Scrapy at scale

Deployment beyond scrapy crawl on your laptop tends to combine Scrapyd for scheduling, Docker for a reproducible environment, and scrapy-redis for sharing a crawl across machines.

Scrapyd

Scrapyd (not a typo, ‘d’ as in httpd or sshd) is a daemon that hosts your spider projects and runs them on demand via HTTP. Install it, deploy your project, kick off crawls with an API call:

pip install scrapyd scrapyd-client
scrapyd  # start the daemon on port 6800
scrapyd-deploy default -p books  # from the project root
curl http://localhost:6800/schedule.json -d project=books -d spider=books

Scrapyd stores logs and stats per crawl and exposes a web UI on port 6800. It’s the shortest path to scheduled crawls for teams with a handful of spiders and no existing container platform.

Docker

A Dockerfile freezes the Python version, Scrapy version, and system libraries. That matters when browser deps (Playwright) or non-Python libraries (lxml, cryptography) get involved.

FROM python:3.13-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["scrapy", "crawl", "books"]

Build, run, and deploy the image anywhere that runs containers (Kubernetes, ECS, Cloud Run). At scale, most teams skip Scrapyd and run Docker directly on their existing platform.

scrapy-redis

For crawls too large for one machine, split the request queue across workers using scrapy-redis. Each spider reads pending requests from a shared Redis queue and writes new requests back to it.

# settings.py
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_URL = "redis://redis-host:6379"

Run 10 identical worker containers pointed at the same Redis instance and they’ll crawl the same domain in parallel without collisions. The deduplication filter guarantees each URL runs once across the whole fleet.

FAQ

Is Scrapy a web crawler or a scraper? Both. The framework handles the whole flow, from fetching pages to following links to extracting structured data. Compare it against BeautifulSoup (parsing only) or requests (HTTP only) to see what “framework” adds.

Is Scrapy better than BeautifulSoup? Different tools. BeautifulSoup parses HTML you already have. Scrapy fetches pages, follows links, manages concurrency, and pipes items to storage. Use BeautifulSoup for a single-page scrape, Scrapy for anything with pagination or multiple pages. A benchmarked comparison shows when each wins on speed and readability.

Is Scrapy free? Yes, BSD-licensed and open source. You can use it in commercial projects without royalties.

Is Scrapy good for large websites? Yes with the right concurrency setting and, once you outgrow one machine, scrapy-redis for distributed crawls. The concurrency benchmark shows the throughput knob works up to server-side throttling limits.

Is Scrapy outdated? No. Scrapy 2.17.0 released in 2025 with active maintenance from the team behind Zyte. Recent versions added native async def support for callbacks and pipelines.

Does Scrapy handle JavaScript out of the box? No. Scrapy fetches HTML and doesn’t run scripts. scrapy-playwright bridges the gap for JS-heavy pages.

Conclusion

Scrapy earns its overhead when the crawl is multi-page and you want structured extraction, concurrent fetching, and pipeline separation without hand-writing an event loop. Ship with CONCURRENT_REQUESTS_PER_DOMAIN = 10 for polite fast crawls, AutoThrottle for unknown thresholds, XPath when selector density matters, and JSONL for streaming exports.

For a single-page scrape, Requests plus BeautifulSoup stays lighter. For heavily JavaScript-rendered sites at volume, offload rendering to the HasData Web Scraping API. For everything in between, the tuned Scrapy setup above holds up.