Web Scraping with Java in 2026

The question “Is Java good for web scraping?” has a complicated reputation. Most tutorials reach for Python or JavaScript, and for throwaway scripts that’s fine (the ecosystem is smaller to navigate). But for scraping hundreds or thousands of pages concurrently, Java 21 is more competitive than its reputation suggests. Virtual threads (Project Loom, shipped in Java 21) handle I/O-bound workloads with less overhead than the per-thread OS approach that made concurrent Java feel expensive.

We measured four approaches fetching 50 pages from books.toscrape.com. Sequential fetching took 15 seconds. ExecutorService with 50 threads took 3.1 seconds. Virtual threads matched that at 3.1 seconds, in one line of code, with no thread pool tuning.

This guide covers Java scraping from a basic Jsoup extractor to concurrent virtual-thread scrapers and Playwright for JS-rendered pages, with every code example compiled and run against a real site.

Is Java Worth It for Web Scraping

Java has earned its “verbose” reputation from J2EE-era patterns. Modern Java is different. Records, text blocks, pattern matching, and the built-in HttpClient (since Java 11) mean the gap with Python has narrowed substantially.

Where Java makes sense for scraping:

• For high-volume concurrent scrapers running 1,000+ simultaneous requests, virtual threads eliminate the thread pool sizing problem that makes Java feel complicated.
• Long-running production scrapers benefit from the JVM’s memory model, mature tooling, and observability ecosystem.
• If your data pipeline is already Java, adding a scraper in the same language avoids a context switch between runtimes.

For one-off data pulls, Python wins. In Python, 10 lines finishes the job before Maven downloads its first dependency.

Setup

Java 21 is the current LTS version. Install it from jdk.java.net or with a package manager:

# macOS
brew install openjdk@21

# Ubuntu / Debian
sudo apt install openjdk-21-jdk

# Windows (winget)
winget install Microsoft.OpenJDK.21

Maven handles dependencies and compilation:

# macOS
brew install maven

# Ubuntu / Debian
sudo apt install maven

# Windows (Chocolatey)
choco install maven

Verify both are installed:

java --version
# openjdk 21.0.12 2024-07-16

mvn --version
# Apache Maven 3.9.16

Create a project directory with this pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>java-scraper</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>21</maven.compiler.source>
        <maven.compiler.target>21</maven.compiler.target>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.jsoup</groupId>
            <artifactId>jsoup</artifactId>
            <version>1.18.3</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>3.3.0</version>
                <configuration>
                    <mainClass>Scraper</mainClass>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Put your Java files in src/main/java/. Compile and run with:

mvn compile
mvn exec:java

The first run downloads dependencies from Maven Central. Subsequent runs compile from cache.

Scraping Static Pages with Jsoup

Jsoup is the standard HTML parsing library for Java. It fetches pages over HTTP, parses the HTML, and lets you query elements with CSS selectors (the same selectors the browser uses in DevTools).

Fetch and parse

Jsoup.connect() fetches a page and returns a parsed Document in one call:

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

Document doc = Jsoup.connect("https://books.toscrape.com/catalogue/page-1.html")
        .userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36")
        .timeout(15_000)
        .get();

Jsoup.connect() makes the HTTP request and returns a parsed Document. The .userAgent() call sets a browser-like User-Agent. Without it, some sites return 403s or bot-detection pages.

Extract elements with CSS selectors

Once you have a Document, .select() queries it with any CSS selector and returns all matching elements as an Elements list:

import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

Elements books = doc.select("article.product_pod");

for (Element book : books) {
    String title  = book.select("h3 > a").attr("title");
    String price  = book.select(".price_color").text();
    String rating = book.select(".star-rating").attr("class")
                        .replace("star-rating ", "");
    String href   = book.select("h3 > a").attr("href");

    System.out.printf("%s — %s (%s)%n", title, price, rating);
}

.select() returns all matching elements as an Elements list. .attr() reads an HTML attribute. .text() returns the visible text content. The selectors are identical to what the browser’s document.querySelectorAll() would accept.

Paginate through multiple pages

Most paginated sites put a “Next” link on each page. Follow it until it disappears:

String pageUrl = "https://books.toscrape.com/catalogue/page-1.html";
String baseUrl = "https://books.toscrape.com/catalogue/";

while (pageUrl != null) {
    Document doc = Jsoup.connect(pageUrl)
            .userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36")
            .timeout(15_000)
            .get();

    for (Element book : doc.select("article.product_pod")) {
        String title = book.select("h3 > a").attr("title");
        String price = book.select(".price_color").text();
        System.out.printf("%s — %s%n", title, price);
    }

    Element next = doc.selectFirst("li.next > a");
    if (next != null) {
        String href = next.attr("href");
        pageUrl = href.startsWith("http") ? href : baseUrl + href;
    } else {
        pageUrl = null;
    }
}

selectFirst("li.next > a") returns null when the last page has no next link, which stops the loop cleanly.

Save to CSV

FileWriter uses the platform’s default charset, which on Windows is usually Windows-1252. Specifying UTF-8 explicitly avoids garbled characters in prices, names, and other non-ASCII content:

import java.io.FileOutputStream;
import java.io.OutputStreamWriter;
import java.io.PrintWriter;
import java.nio.charset.StandardCharsets;

try (PrintWriter csv = new PrintWriter(
        new OutputStreamWriter(
                new FileOutputStream("books.csv"),
                StandardCharsets.UTF_8))) {

    csv.println("title,price,rating,url");

    for (Element book : doc.select("article.product_pod")) {
        String title  = book.select("h3 > a").attr("title");
        String price  = book.select(".price_color").text();
        String rating = book.select(".star-rating").attr("class")
                            .replace("star-rating ", "");
        String url    = baseUrl + book.select("h3 > a").attr("href")
                                      .replace("../", "");

        csv.printf("\"%s\",\"%s\",\"%s\",\"%s\"%n", title, price, rating, url);
    }
}

Running this against books.toscrape.com produces a CSV with 60 books across 3 pages, each with title, price, rating, and URL.

Concurrent Scraping with Virtual Threads

Fetching pages one at a time is slow. Each request blocks while waiting for the server to respond. At 300 ms average per page, 50 pages takes 15 seconds even if your parsing takes milliseconds. The fix is to fire requests concurrently.

Java developers have been using extends Thread for this for decades:

// Old pattern — don't do this
public class FetchThread extends Thread {
    public void run() {
        // fetch one page
    }
}

FetchThread t1 = new FetchThread(url1);
FetchThread t2 = new FetchThread(url2);
t1.start();
t2.start();

The problem is scaling. Each Thread maps to one OS thread, which costs around 1 MB of stack memory. Spawning 5,000 threads for 5,000 pages exhausts the OS thread limit. The practical ceiling is a few hundred.

ExecutorService with a fixed thread pool was the next step:

// 2010s pattern — reasonable, but needs pool sizing
ExecutorService pool = Executors.newFixedThreadPool(50);
pool.submit(() -> fetchPage(url));

This reuses 50 threads for all tasks, which is safer. But you still have to choose the pool size. A pool that’s too small leaves throughput on the table. Too large and you overload the server.

Java 21 virtual threads remove the sizing question. Virtual threads are scheduled by the JVM, not the OS. Blocking one while a network request is in flight doesn’t block the underlying OS thread. The JVM parks the virtual thread and runs another. You can create millions of them.

try (ExecutorService pool = Executors.newVirtualThreadPerTaskExecutor()) {
    // one virtual thread per task, no pool sizing needed
}

Benchmark: all four approaches on 50 pages

We ran all four approaches fetching 50 pages from books.toscrape.com on Java 21.0.12 / Windows 11:

Sequential takes 15 seconds. All three concurrent approaches finish in around 3 seconds, a 5× improvement, because network latency dominates and all three overlap that wait time. Virtual threads don’t outperform ExecutorService at 50 pages because 50 tasks fit comfortably into 50 OS threads. At 5,000 pages that changes. ExecutorService(50) still processes 50 at a time while virtual threads run all 5,000 concurrently, with no code change required.

Full virtual threads scraper

Here’s a complete scraper that fetches 10 pages concurrently and collects results in submission order:

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

public class ConcurrentScraper {

    static final String BASE_URL = "https://books.toscrape.com/catalogue/page-%d.html";
    static final int PAGES = 10;

    record Book(String title, String price, String rating) {}

    static List<Book> scrapePage(int page) throws Exception {
        String url = String.format(BASE_URL, page);
        Document doc = Jsoup.connect(url)
                .userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36")
                .timeout(15_000)
                .get();

        List<Book> books = new ArrayList<>();
        for (Element el : doc.select("article.product_pod")) {
            books.add(new Book(
                    el.select("h3 > a").attr("title"),
                    el.select(".price_color").text(),
                    el.select(".star-rating").attr("class").replace("star-rating ", "")
            ));
        }
        return books;
    }

    public static void main(String[] args) throws Exception {
        long start = System.nanoTime();

        List<Callable<List<Book>>> tasks = new ArrayList<>();
        for (int page = 1; page <= PAGES; page++) {
            final int p = page;
            tasks.add(() -> scrapePage(p));
        }

        List<List<Book>> results;
        try (ExecutorService pool = Executors.newVirtualThreadPerTaskExecutor()) {
            List<Future<List<Book>>> futures = pool.invokeAll(tasks);
            results = new ArrayList<>();
            for (Future<List<Book>> f : futures) results.add(f.get());
        }

        double elapsed = (System.nanoTime() - start) / 1e9;
        int total = results.stream().mapToInt(List::size).sum();
        System.out.printf("Scraped %d books from %d pages in %.2f seconds%n",
                total, PAGES, elapsed);
    }
}

This scrapes 200 books from 10 pages in under 2 seconds. The try-with-resources on ExecutorService shuts the pool down automatically when all tasks complete (AutoCloseable on ExecutorService, added in Java 19), so no manual pool.shutdown() or pool.awaitTermination() call needed.

pool.invokeAll() returns results in submission order, not completion order, so page 1’s results always come first regardless of which network request finished first.

Custom Headers and Retry Logic

Jsoup handles User-Agent through its builder API. For fine-grained control over all request headers, timeouts, and retry behavior, Java’s built-in HttpClient (available since Java 11, no extra dependency) is the right choice.

Setting browser-like headers

Build a request with HttpRequest.newBuilder() and attach headers with .header() before sending:

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;

HttpClient client = HttpClient.newBuilder()
        .connectTimeout(Duration.ofSeconds(10))
        .followRedirects(HttpClient.Redirect.NORMAL)
        .build();

HttpRequest request = HttpRequest.newBuilder()
        .uri(URI.create("https://books.toscrape.com/catalogue/page-1.html"))
        .timeout(Duration.ofSeconds(15))
        .header("User-Agent",
                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) " +
                "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36")
        .header("Accept-Language", "en-US,en;q=0.9")
        .header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
        .header("Referer", "https://www.google.com/")
        .build();

HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
String html = response.body();

Accept-Language and Referer make requests look more like a real browser origin. Some sites check these together with User-Agent, and how you rotate them affects block rates at scale.

Retry with exponential backoff

Rate limits, brief server overload, and timeouts are all normal at scale. A retry loop handles most of them:

static String fetchWithRetry(HttpClient client, String url, int maxAttempts)
        throws Exception {

    long delayMs = 1_000;

    for (int attempt = 1; attempt <= maxAttempts; attempt++) {
        HttpRequest req = HttpRequest.newBuilder()
                .uri(URI.create(url))
                .timeout(Duration.ofSeconds(15))
                .header("User-Agent",
                        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) " +
                        "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36")
                .build();

        try {
            HttpResponse<String> resp = client.send(req, HttpResponse.BodyHandlers.ofString());
            if (resp.statusCode() == 200) return resp.body();
            System.err.printf("attempt %d: status %d%n", attempt, resp.statusCode());
        } catch (Exception e) {
            System.err.printf("attempt %d: %s%n", attempt, e.getMessage());
        }

        if (attempt < maxAttempts) {
            Thread.sleep(delayMs);
            delayMs *= 2;
        }
    }

    throw new RuntimeException("Failed after " + maxAttempts + " attempts: " + url);
}

The delay doubles on each retry (1 s, 2 s, 4 s), backing off from the server rather than hammering it after a 429. Wrap this inside your virtual-thread task and you get per-request retry with no shared state issues.

Proxy configuration

HttpClient supports proxies through ProxySelector:

import java.net.InetSocketAddress;
import java.net.ProxySelector;
import java.net.Authenticator;
import java.net.PasswordAuthentication;

HttpClient clientWithProxy = HttpClient.newBuilder()
        .proxy(ProxySelector.of(new InetSocketAddress("proxy.example.com", 8080)))
        .authenticator(new Authenticator() {
            @Override
            protected PasswordAuthentication getPasswordAuthentication() {
                return new PasswordAuthentication("username", "password".toCharArray());
            }
        })
        .build();

Requests made with this client route through the proxy automatically. No wrapper libraries needed.

Scraping JS-Rendered Pages with Playwright

Jsoup and HttpClient fetch the raw HTML returned by the server. Pages that render their content with JavaScript (dashboards, single-page apps, infinite scroll feeds) need a real browser to execute that JavaScript before you can extract anything meaningful.

Playwright for Java drives Chromium, Firefox, or WebKit headlessly. It’s the modern choice over both HtmlUnit (which simulates a browser in software and has poor compatibility with modern JS frameworks) and Selenium (which still works but requires separate WebDriver version management).

Add the dependency to pom.xml:

<dependency>
    <groupId>com.microsoft.playwright</groupId>
    <artifactId>playwright</artifactId>
    <version>1.47.0</version>
</dependency>

On first run, Playwright downloads the browser binaries automatically. No separate ChromeDriver to match against your installed Chrome version.

import com.microsoft.playwright.*;
import com.microsoft.playwright.options.WaitUntilState;

import java.util.List;

public class PlaywrightExample {

    public static void main(String[] args) {
        try (Playwright playwright = Playwright.create()) {
            Browser browser = playwright.chromium().launch(
                    new BrowserType.LaunchOptions().setHeadless(true)
            );

            BrowserContext context = browser.newContext(
                    new Browser.NewContextOptions()
                            .setUserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) " +
                                    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36")
            );

            Page page = context.newPage();
            page.navigate(
                    "https://books.toscrape.com/catalogue/page-1.html",
                    new Page.NavigateOptions().setWaitUntil(WaitUntilState.DOMCONTENTLOADED)
            );

            List<String> titles = page.locator("article.product_pod h3 > a")
                    .allInnerTexts();
            List<String> prices = page.locator("article.product_pod .price_color")
                    .allInnerTexts();

            for (int i = 0; i < titles.size(); i++) {
                System.out.printf("%s — %s%n", titles.get(i), prices.get(i));
            }

            browser.close();
        }
    }
}

WaitUntilState.DOMCONTENTLOADED waits until the DOM is parsed before extracting. For pages that load content after the initial render, use WaitUntilState.NETWORKIDLE or wait for a specific element with page.waitForSelector(".content-loaded").

The try-with-resources block ensures the browser process shuts down even if extraction throws. An orphaned Chromium process keeps consuming memory, so this matters.

Playwright also handles clicking, form submission, and scrolling. For pure data extraction from static pages, Jsoup is faster and simpler. Use Playwright when you need JavaScript execution or browser interaction.

Web Scraping with HasData

The HasData Web Scraping API fetches pages server-side and returns structured data or raw HTML. Session management, proxy rotation, and the rest of the request infrastructure run on the API’s side. Add org.json to pom.xml for JSON parsing:

<dependency>
    <groupId>org.json</groupId>
    <artifactId>json</artifactId>
    <version>20240303</version>
</dependency>

Post a request with extractRules to pull structured data directly without parsing HTML yourself:

import org.json.JSONArray;
import org.json.JSONObject;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;

public class HasDataExample {

    static final String API_URL = "https://api.hasdata.com/scrape/web";
    static final String API_KEY = System.getenv("HASDATA_API_KEY");

    public static void main(String[] args) throws Exception {
        String body = new JSONObject()
                .put("url", "https://books.toscrape.com/catalogue/page-1.html")
                .put("extractRules", new JSONObject()
                        .put("titles",  "article.product_pod h3 > a @title")
                        .put("prices",  "article.product_pod .price_color")
                        .put("ratings", "article.product_pod .star-rating @class"))
                .toString();

        HttpClient client = HttpClient.newBuilder()
                .connectTimeout(Duration.ofSeconds(15))
                .build();

        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create(API_URL))
                .timeout(Duration.ofSeconds(30))
                .header("Content-Type", "application/json")
                .header("x-api-key", API_KEY)
                .POST(HttpRequest.BodyPublishers.ofString(body))
                .build();

        HttpResponse<String> response = client.send(
                request, HttpResponse.BodyHandlers.ofString());

        JSONObject json     = new JSONObject(response.body());
        JSONObject data     = json.getJSONObject("extractedData");
        JSONArray  titles   = data.getJSONArray("titles");
        JSONArray  prices   = data.getJSONArray("prices");
        JSONArray  ratings  = data.getJSONArray("ratings");

        for (int i = 0; i < titles.length(); i++) {
            String rating = ratings.getString(i).replace("star-rating ", "");
            System.out.printf("%s — %s (%s)%n",
                    titles.getString(i), prices.getString(i), rating);
        }
    }
}

extractRules maps a name to a CSS selector. Append @attribute-name to extract an attribute rather than text content. @title reads the title attribute and @href gets the link URL. The API returns all matches as a JSON array under extractedData.

Set your API key as an environment variable before running:

export HASDATA_API_KEY=your_key_here
mvn exec:java

The calling code is the same for any target site. Session management, proxy rotation, and request configuration are handled on the API’s side.

Web Crawling

A crawler follows links from page to page, building a map of a site. The core loop maintains two sets: URLs left to visit and URLs already visited.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;

import java.util.ArrayDeque;
import java.util.Deque;
import java.util.HashSet;
import java.util.Set;

public class Crawler {

    static final String START_URL = "https://books.toscrape.com/";
    static final String DOMAIN    = "books.toscrape.com";
    static final int    MAX_PAGES = 20;

    public static void main(String[] args) throws Exception {
        Deque<String> queue   = new ArrayDeque<>();
        Set<String>   visited = new HashSet<>();

        queue.add(START_URL);

        while (!queue.isEmpty() && visited.size() < MAX_PAGES) {
            String url = queue.poll();
            if (visited.contains(url)) continue;
            visited.add(url);

            System.out.println("Crawling: " + url);

            Document doc = Jsoup.connect(url)
                    .userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36")
                    .timeout(15_000)
                    .get();

            for (Element book : doc.select("article.product_pod")) {
                String title = book.select("h3 > a").attr("title");
                String price = book.select(".price_color").text();
                System.out.printf("  %s — %s%n", title, price);
            }

            for (Element link : doc.select("a[href]")) {
                String abs = link.absUrl("href");
                if (abs.contains(DOMAIN) && !visited.contains(abs)) {
                    queue.add(abs);
                }
            }
        }

        System.out.printf("%nCrawled %d pages%n", visited.size());
    }
}

link.absUrl("href") resolves relative URLs against the current page’s base, turning ../catalogue/page-2.html into the full URL automatically. The MAX_PAGES guard keeps the crawler from running indefinitely during development.

To run this concurrently, the visited set needs thread safety. Swap HashSet for ConcurrentHashMap.newKeySet():

Set<String> visited = ConcurrentHashMap.newKeySet();

Wrap the fetch and enqueue logic in a virtual-thread task and the crawler runs concurrent page fetches without any other changes to the structure.

Java Scraping Libraries

The table below compares the five libraries covered in this guide by use case, JavaScript support, and whether they require an external dependency.

Jsoup and HttpClient cover most scraping tasks. Add Playwright when a page requires JavaScript. Selenium is a reasonable fallback if you already have it in your stack, but Playwright’s auto-managed browsers and cleaner API make it the better default for new projects. HtmlUnit simulates JavaScript without launching a real browser, which sounds attractive until you hit a site using any modern JS framework. Compatibility with current frameworks is limited.

Conclusion

Java’s concurrency story changed materially with Java 21. Virtual threads reduce a concurrent I/O scraper to one executor line and no pool sizing decisions. The code is simpler than a tuned ExecutorService, and it scales to workloads where OS threads would run out of headroom. The benchmark on 50 pages shows the 5× gain over sequential. At 5,000 pages, the gap between a fixed pool and virtual threads grows much further.

The practical stack looks like this: Jsoup for static HTML, HttpClient when you need header control or retry logic, Playwright for JS-rendered pages, and HasData’s API when anti-bot protection blocks everything else. The extends Thread pattern that most older Java scraping tutorials still teach was the right approach in 2004, before ExecutorService and virtual threads existed.