Web Scraping with Rust in 2026

The compiler rejects data races at build time. In a threaded Python scraper, a race between two fetches that share state crashes at runtime or corrupts data. In Rust, that code does not compile. For a scraper running 50 concurrent tasks against 5 000 pages, concurrent scraping in Rust is correct by construction.

The first cargo build takes 40-60 seconds and the borrow checker resists shared mutable state patterns that work in Python and Node.js. Browser automation crates are functional but have fewer maintained examples than Puppeteer or Playwright. If you need a scraper this afternoon, Python or Node.js gets you there faster.

For throughput-heavy workloads, Rust is worth the setup cost. Fetching 50 pages with tokio::spawn takes 3 seconds at 10 workers and under 1 second at 50. The same loop with blocking requests takes 34 seconds, and on a 5 000-page crawl that adds up to 5 minutes versus nearly an hour.

The HasData Web Scraping API handles JavaScript rendering and proxy rotation server-side, returning HTML or structured JSON to any reqwest call.

Installation

Rust installs through rustup. On Linux or macOS:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

On Windows, download the installer from rust-lang.org. When the prompt appears, select option 1 for the default toolchain.

Create a new project with cargo new rust-scraper, then declare all dependencies in Cargo.toml before writing any code:

[package]
name = "rust-scraper"
version = "0.1.0"
edition = "2021"

[dependencies]
reqwest         = { version = "0.12", features = ["blocking", "json"] }
scraper         = "0.22"
tokio           = { version = "1", features = ["full"] }
headless_chrome = "1.0"
serde           = { version = "1", features = ["derive"] }
serde_json      = "1"
csv             = "1"

Run cargo build once to fetch and compile dependencies. The first build takes a few minutes. Subsequent builds reuse cached artifacts.

Rust crates for web scraping

Most Rust scrapers combine reqwest for HTTP, scraper for HTML parsing, and tokio for concurrent fetches. Browser crates come in only when the target page renders content in JavaScript.

Add a browser crate only when the page renders its content in JavaScript on the client side.

Scraping static pages

The example below scrapes a product listing from demo.opencart.com, collecting the title, URL, description, and prices for each product.

The full main.rs:

use reqwest::blocking::Client;
use scraper::{Html, Selector};
use csv::Writer;

struct Product {
    title:       String,
    url:         String,
    description: String,
    price_new:   String,
    price_tax:   String,
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client   = Client::new();
    let response = client.get("https://demo.opencart.com/").send()?;
    let body     = response.text()?;

    let document  = Html::parse_document(&body);
    let item_sel  = Selector::parse("div.col").unwrap();
    let title_sel = Selector::parse(".description h4").unwrap();
    let url_sel   = Selector::parse("h4 a").unwrap();
    let desc_sel  = Selector::parse(".description p").unwrap();
    let price_sel = Selector::parse("span.price-new").unwrap();
    let tax_sel   = Selector::parse("span.price-tax").unwrap();

    let mut products: Vec<Product> = Vec::new();

    for item in document.select(&item_sel) {
        let title = item.select(&title_sel).next()
            .map(|e| e.text().collect::<String>())
            .unwrap_or_default();
        let url = item.select(&url_sel).next()
            .and_then(|a| a.value().attr("href"))
            .unwrap_or("").to_string();
        let description = item.select(&desc_sel).next()
            .map(|e| e.text().collect::<String>())
            .unwrap_or_default();
        let price_new = item.select(&price_sel).next()
            .map(|e| e.text().collect::<String>())
            .unwrap_or_default();
        let price_tax = item.select(&tax_sel).next()
            .map(|e| e.text().collect::<String>())
            .unwrap_or_default();

        if !title.is_empty() {
            products.push(Product { title, url, description, price_new, price_tax });
        }
    }

    let mut writer = Writer::from_path("products.csv")?;
    writer.write_record(["title", "url", "description", "price_new", "price_tax"])?;
    for p in &products {
        writer.write_record([&p.title, &p.url, &p.description, &p.price_new, &p.price_tax])?;
    }
    writer.flush()?;
    println!("Saved {} products", products.len());
    Ok(())
}

Html::parse_document builds the parse tree from the raw HTML string. Selector::parse compiles a CSS selector once. Calling it inside a tight loop adds measurable overhead on large pages. Attributes come out through .value().attr("name"), inner text through .text().collect::<String>().

Run with cargo run. The result appears in products.csv alongside the binary.

Each product maps to one CSV row, with an empty string for any field the selector did not match on that element.

Concurrent scraping with tokio

The case for Rust in scraping is concurrency without data races. Rust’s ownership model makes data races impossible at compile time, which removes a class of bugs that only surface under load in Python and Node.js. tokio is the standard async runtime. reqwest::Client is designed to be cloned across tasks because it holds a connection pool behind an Arc, so cloning it is cheap and both clones share the same open connections.

The example below fetches 50 pages of books.toscrape.com concurrently using tokio::spawn and JoinSet:

use reqwest::Client;
use scraper::{Html, Selector};
use tokio::task::JoinSet;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::new();
    let mut set = JoinSet::new();

    for page in 1..=50 {
        let client = client.clone();
        let url = format!(
            "https://books.toscrape.com/catalogue/page-{}.html",
            page
        );
        set.spawn(async move {
            client.get(&url).send().await?.text().await
        });
    }

    let title_sel = Selector::parse("h3 a").unwrap();
    let mut total = 0usize;

    while let Some(result) = set.join_next().await {
        if let Ok(Ok(body)) = result {
            let doc = Html::parse_document(&body);
            total += doc.select(&title_sel).count();
        }
    }

    println!("Found {} books across 50 pages", total);
    Ok(())
}

Fetching all 50 pages sequentially takes 34 seconds. At 10 concurrent workers the same fetch completes in 3 seconds, and at 50 workers, under 1 second. On a 5 000-page crawl those numbers translate to 5 minutes versus nearly an hour.

JoinSet drives the concurrency without channels or mutexes. .join_next() yields each task’s result as it completes, so you can start parsing completed responses while remaining fetches are still in flight.

Scraping JavaScript-rendered pages

headless_chrome connects to Chrome over the DevTools Protocol. No separate ChromeDriver binary is needed. The crate launches a Chrome process directly, which eliminates the version-matching problem that Selenium setups require.

Chrome or Chromium must be installed on the system. The example below navigates to a page and extracts product titles after the page finishes loading:

use headless_chrome::Browser;
use scraper::{Html, Selector};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let browser = Browser::default()?;
    let tab = browser.new_tab()?;
    tab.navigate_to("https://demo.opencart.com/")?;
    tab.wait_until_navigated()?;

    let html = tab.get_content()?;
    let document = Html::parse_document(&html);

    let title_sel = Selector::parse(".description h4").unwrap();
    for title in document.select(&title_sel) {
        println!("{}", title.text().collect::<String>());
    }

    Ok(())
}

tab.wait_until_navigated() blocks until the page’s load event fires. For content that appears on a timer or after user interaction, call tab.wait_for_element("selector") to block until a specific element exists in the DOM.

Two alternatives cover cases where headless_chrome falls short. chromiumoxide also speaks the DevTools Protocol but runs fully async, so it fits naturally into a tokio pipeline. Browser tasks can share the same JoinSet as reqwest fetches, which is not possible with headless_chrome’s blocking API. thirtyfour speaks the WebDriver protocol instead of DevTools. Use it when you need Chrome, Firefox, and Edge under one API, or when your team already runs a Selenium Grid. Both crates are async and integrate with tokio the same way.

Using the HasData Web Scraping API

The HasData Web Scraping API fetches the page, executes JavaScript, and returns the result. Proxy rotation and ban detection happen server-side. The Rust code only makes an HTTP call.

The extractRules parameter maps output field names to CSS selectors. The API returns the extracted values under extractedData:

use reqwest::blocking::Client;
use serde_json::{json, Value};
use std::env;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let api_key = env::var("HASDATA_API_KEY")?;

    let body = json!({
        "url": "https://demo.opencart.com/",
        "extractRules": {
            "title": ".description h4",
            "price": "span.price-new",
            "tax":   "span.price-tax"
        }
    });

    let client = Client::new();
    let response = client
        .post("https://api.hasdata.com/scrape/web")
        .header("x-api-key", api_key.as_str())
        .json(&body)
        .send()?;

    let json: Value = response.json()?;
    if let Some(extracted) = json.get("extractedData") {
        println!("{}", serde_json::to_string_pretty(extracted)?);
    }

    Ok(())
}

Read the API key from an environment variable rather than hardcoding it. The extractRules keys become the field names in extractedData, and the values are standard CSS selectors. Before calling the API, check whether the target data appears in view-source:. If it does, reqwest alone handles it without any API credits.

Web crawling

A crawler follows links recursively. The example below starts at demo.opencart.com and collects every internal link, using a HashSet to skip URLs already visited:

use reqwest::blocking::Client;
use scraper::{Html, Selector};
use std::collections::HashSet;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::new();
    let start  = "https://demo.opencart.com/";
    let mut visited = HashSet::new();
    crawl(&client, start, &mut visited);
    println!("Crawled {} pages", visited.len());
    Ok(())
}

fn crawl(client: &Client, url: &str, visited: &mut HashSet<String>) {
    if visited.contains(url) {
        return;
    }
    visited.insert(url.to_string());

    let Ok(response) = client.get(url).send() else { return };
    let Ok(body)     = response.text()        else { return };

    let document = Html::parse_document(&body);
    let link_sel = Selector::parse("a[href]").unwrap();

    for element in document.select(&link_sel) {
        if let Some(href) = element.value().attr("href") {
            if href.starts_with("http") && href.contains("demo.opencart.com") {
                println!("{}", href);
                crawl(client, href, visited);
            }
        }
    }
}

Client is passed by reference so every recursive call shares the same connection pool. The HashSet prevents revisiting pages and cuts off infinite loops on sites with circular links. Selector::parse("a[href]") recompiles on each crawl call. Move it above the recursion if the site is large enough to make that cost visible in profiling. For production crawlers, the spider crate handles politeness delays, concurrency limits, and sitemap parsing without building those pieces from scratch.

When to use Rust for web scraping

Rust performs well on workloads bound by network I/O or CPU-heavy parsing, and where the compilation overhead and steeper learning curve are acceptable costs.

The compiler rejects data races at build time, which removes a class of bugs that only appear under load in Python and Node.js. For a scraper that runs infrequently or on a handful of pages, that guarantee does not justify a 40-second compile time and the additional code the borrow checker requires.

Rust makes practical sense when you are processing millions of pages and memory or CPU is the constraint, when scraping logic lives inside a larger Rust system, or when you need a static binary that deploys without installing a runtime on the target machine. The scraper and reqwest crates are stable and well-maintained. The browser automation crates (headless_chrome, chromiumoxide, thirtyfour) are functional but have fewer users than Puppeteer or Playwright, which means fewer examples and slower bug fixes in the open ecosystem.