Web Scraping in R in 2026 (Benchmarked)

Web scraping in R is fetching HTML with rvest, hitting hidden APIs with httr2, driving Chrome with chromote when a page needs JavaScript, and wrapping crawls in polite for rate limits. The CSS layer in rvest hides a translation cost. It converts every CSS query to XPath through the selectr package before running it. On a 5-book page, XPath extraction runs 64% faster than CSS. On 200 books the gap closes to 10% as per-element extraction dominates.

I benchmarked the stack on books.toscrape.com and quotes.toscrape.com. chromote takes 4.8 seconds to boot Chrome the first time, then matches read_html at 1.2 seconds per fetch. read_html on a JavaScript-rendered page returns 0 rows and no error.

chromote replaced RSelenium as the default JavaScript scraper in 2024. RSelenium still works but relies on Java and Selenium.

R 4.6.1, rvest 1.0.5, chromote 0.5.1.

What R is good for in web scraping

R turns HTML into a data frame in five lines. read_html fetches the page, html_elements pulls the pieces you want, and data.frame() or tibble() collects them into a structure that plugs straight into dplyr, ggplot2, and the rest of the tidyverse. Python and Node give you parsed elements you still have to marshal into a DataFrame or array.

Concurrency in R is thin. There is no async runtime like Python’s asyncio or Node’s event loop. Parallel HTTP through the future and furrr packages works but adds complexity that Scrapy or Crawlee handle out of the box.

Choose R when the scraper is one step in a larger data pipeline and the target is under a few hundred pages. Python or Node scale better when the scraper is the pipeline and you need to hit tens of thousands of pages concurrently.

The modern R scraping stack

Which of the four packages to use depends on how the target page delivers its data.

rvest

rvest is the canonical scraping library in R. It wraps the xml2 package (which binds to libxml2 in C) with a tidyverse-flavored API. read_html fetches, html_elements selects, html_text2 extracts, html_attr reads attributes. All results are xml_nodeset objects that pipe cleanly.

If the target site returns HTML on the initial GET, rvest is the whole toolchain.

httr2

httr2 is the tidy successor to httr. Builds requests as first-class objects with request(), sends them with req_perform(). Built-in retries via req_retry(), rate limiting via req_throttle(), JSON body parsing via resp_body_json().

httr2 wins over rvest when the browser’s Network tab shows the page loads its data through a JSON API. Hitting that API directly is faster and cleaner than parsing rendered HTML.

chromote

chromote is a headless Chrome client for R. Uses Chrome DevTools Protocol (CDP), the same protocol Playwright and Puppeteer use. Replaced RSelenium as the default JavaScript scraper in 2024 by dropping the Java and Selenium dependency.

Also powers rvest::read_html_live(), which gives you an rvest-shaped API over a chromote session. Use read_html_live() for one-off JS scraping and chromote directly when you need explicit browser control (waits, page interaction, network interception).

RSelenium and Rcrawler still work. RSelenium uses Selenium 4 with WebDriver protocol and requires a Java runtime. Rcrawler is a full crawling framework with parallel workers. Both remain in maintenance and are fine on existing codebases, but new projects starting in 2024 or later default to chromote and polite.

Your first R scraper

Install the packages once from your R console or RStudio. The rest of the section uses them.

install.packages(c("rvest", "dplyr", "readr", "purrr"))

Pull book titles, prices, and ratings from a books.toscrape.com listing page. tibble::tibble() builds a tidyverse-flavored data frame with cleaner printing and no string-to-factor conversion. The base data.frame() is a drop-in replacement if you want to skip the tidyverse dependency:

library(rvest)

url <- "https://books.toscrape.com/"
page <- read_html(url)
articles <- html_elements(page, "article.product_pod")

books <- tibble::tibble(
  title  = html_attr(html_element(articles, "h3 a"), "title"),
  price  = html_text2(html_element(articles, "p.price_color")),
  rating = sub("star-rating\\s+", "", html_attr(html_element(articles, "p.star-rating"), "class"))
)

head(books, 3)

The first three rows come back as a tibble with title, price, and rating columns:

# A tibble: 3 × 3
  title                price  rating
  <chr>                <chr>  <chr>
1 A Light in the Attic £51.77 Three
2 Tipping the Velvet   £53.74 One
3 Soumission           £50.10 One

read_html fetches and parses the page. html_elements returns a nodeset with one <article> per book. html_element (singular) picks the first match inside each parent, and html_attr / html_text2 extract attributes and text. The result is a tibble with 20 rows.

For multi-page scrapes, iterate with purrr::map_dfr, which row-binds each tibble into one result:

library(purrr)

urls <- sprintf("https://books.toscrape.com/catalogue/page-%d.html", 1:3)

all_books <- map_dfr(urls, function(page_url) {
  arts <- read_html(page_url) |> html_elements("article.product_pod")
  tibble::tibble(
    title  = html_attr(html_element(arts, "h3 a"), "title"),
    price  = html_text2(html_element(arts, "p.price_color")),
    rating = sub("star-rating\\s+", "", html_attr(html_element(arts, "p.star-rating"), "class"))
  )
})

Three pages of books.toscrape.com produce a 60-row tibble.

Save it to CSV with readr::write_csv:

library(readr)
write_csv(all_books, "books.csv")

The tibble also goes straight into dplyr without a conversion step. Average price per rating in three lines:

library(dplyr)

all_books |>
  mutate(price_num = as.numeric(sub("£", "", price))) |>
  group_by(rating) |>
  summarise(n = n(), avg_price = round(mean(price_num), 2)) |>
  arrange(desc(avg_price))

Grouped by rating, the sample shows three-star books averaging highest and five-star averaging lowest:

# A tibble: 5 × 3
  rating     n avg_price
  <chr>  <int>     <dbl>
1 Three     13      38.8
2 Four      10      35.5
3 One       15      34.3
4 Two        8      33.9
5 Five      14      32.5

The whole pipeline runs inside R without a conversion step between HTML and grouped output.

CSS vs XPath in R, benchmarked

rvest supports both CSS selectors and XPath expressions through html_elements. The same book titles extracted two ways:

library(rvest)

page <- read_html("https://books.toscrape.com/")

# CSS
titles_css <- page |>
  html_elements("article.product_pod h3 a") |>
  html_attr("title")

# XPath
titles_xp <- page |>
  html_elements(xpath = "//article[contains(@class, 'product_pod')]//h3/a") |>
  html_attr("title")

CSS syntax is shorter for class-based selectors. XPath wins for text() predicates, ancestor or following-sibling axes, and nested class matching. //p[contains(text(), 'In stock')] has no clean CSS equivalent short of :has(), which the underlying selectr stack supports only partially.

Tutorials default to CSS, but the runtime cost tells a different story.

Benchmark setup

• Synthetic HTML with N book articles (5, 10, 20, 50, 100, 200) to control page size
• Extraction: title, price, and rating class per article
• 500 iterations each style, 50-iteration warmup
• microbenchmark for measurement, R 4.6.1, rvest 1.0.5

Results


The gap comes from the selectr package. Every CSS query in rvest is translated to XPath before running, and that translation is a fixed cost per call. On a 5-book page the translation dominates (0.36× ratio). On 200 books per-element extraction takes over and the gap closes to 0.90×.

For most real listing pages (20-50 items), XPath runs 25-40% faster than CSS.

Selector guidance by situation:

Default to XPath in rvest, and use CSS as a copy-paste target from SelectorGadget.

Handling HTTP with httr2

When a page’s data loads via JavaScript, the DevTools Network tab often shows an XHR pulling JSON. Hitting that endpoint directly with httr2 skips the browser and returns cleaner data than parsing rendered HTML. Open the target page in Chrome DevTools, filter Network to XHR, find the JSON request, copy its URL, and plug it into request().


The basic pattern builds a request, performs it, and parses the JSON response:

library(httr2)

resp <- request("https://jsonplaceholder.typicode.com/posts") |>
  req_url_query(userId = 1) |>
  req_headers(`User-Agent` = "MyScraper/1.0") |>
  req_perform()

posts <- resp_body_json(resp)

request() builds a request object. Every req_* function returns a modified request, so they chain with the pipe. req_perform() sends it. resp_body_json() parses the response body as JSON, returning an R list.

Retries and rate limiting

httr2 has retry and throttle logic built in:

resp <- request("https://api.example.com/products") |>
  req_throttle(rate = 2, realm = "api.example.com") |>
  req_retry(max_tries = 3, retry_on_failure = TRUE) |>
  req_perform()

req_throttle(rate = 2) caps outgoing requests to 2 per second per realm. The realm groups requests that share a limit. req_retry(max_tries = 3, retry_on_failure = TRUE) retries transient failures (5xx, 429, network errors) with exponential backoff, up to three attempts.

Sessions and cookies

For sites that require login or track state across requests, req_cookie_preserve() writes cookies to a shared jar file. Every request that references the same file reads and writes to it.

cookie_jar <- tempfile()

# Log in and save cookies to the jar file
login <- request("https://example.com/login") |>
  req_body_form(username = "user", password = Sys.getenv("EXAMPLE_PASS")) |>
  req_cookie_preserve(cookie_jar) |>
  req_perform()

# Subsequent requests reuse the same jar automatically
data <- request("https://example.com/dashboard") |>
  req_cookie_preserve(cookie_jar) |>
  req_perform() |>
  resp_body_json()

For sites that use CSRF tokens, dynamic form parameters, or Cloudflare-style challenges, httr2 alone falls short. Either drive a real browser through chromote or offload the harder cases to a scraping API.

Dynamic pages with chromote and read_html_live

read_html returns whatever HTML the server sends. If the page renders its content through JavaScript after load, the raw HTML has empty placeholders and read_html extracts nothing. chromote runs the page in a real Chrome instance, waits for JavaScript to execute, and returns the fully rendered HTML.

The rvest-shaped wrapper is read_html_live. Same API as read_html, chromote inside:

library(rvest)

sess <- read_html_live("https://quotes.toscrape.com/js/")
quotes <- sess |>
  html_elements("div.quote span.text") |>
  html_text2()

length(quotes)

That returns 10 quotes. The same URL through read_html:

read_html("https://quotes.toscrape.com/js/") |>
  html_elements("div.quote span.text") |>
  html_text2() |>
  length()

An empty character vector. rvest fetched the page, found the CSS didn’t match, and returned without an error.

For explicit browser control (waits, clicks, network interception), use chromote directly and feed the rendered HTML into rvest:

library(chromote)
library(rvest)

sess <- ChromoteSession$new()
sess$Page$navigate("https://quotes.toscrape.com/js/")
sess$Page$loadEventFired()

html <- sess$Runtime$evaluate("document.documentElement.outerHTML")$result$value
sess$close()

quotes <- read_html(html) |>
  html_elements("div.quote span.text") |>
  html_text2()

Works the same, but you now have sess for calling sess$Page$navigate, sess$Input$dispatchKeyEvent, sess$Network$enable, and the full Chrome DevTools Protocol.

Benchmark setup

• Static site: books.toscrape.com/catalogue/page-1.html
• JS site: quotes.toscrape.com/js/
• Static bench: 15 rounds per method, chromote round 1 is cold-start
• JS bench: 5 rounds per method for correctness comparison
• R 4.6.1, rvest 1.0.5, chromote 0.5.1, Chrome via CDP

Results

Speed on a static site (books.toscrape.com):


Correctness on a JS-rendered site (quotes.toscrape.com/js/), 5 rounds each:

Speed comparison on the JS site is meaningless because read_html completes in under a second by fetching an empty page. read_html_live takes longer but returns actual data.

The cold-start on chromote is a one-time cost for the Chrome process. For a single-page scrape from an R script that exits, budget five seconds. For a long-running scrape that reuses one session, per-fetch time matches static read_html.

RSelenium is the legacy alternative. It drives Selenium 4 through WebDriver and requires Java plus a browser driver on the machine. Chromote is faster to boot (no Java startup) and simpler to install. New projects starting in 2024 or later use chromote.

For JavaScript-heavy sites at scale (hundreds of pages per day or more), offload rendering to the HasData Web Scraping API or a similar service. Chromote’s memory footprint compounds at scale, and each Chrome instance eats CPU.

Polite web scraping

polite wraps rvest and httr2 with etiquette rules. A polite session starts by introducing itself to the site with bow(), which reads the site’s robots.txt, checks whether the target path is allowed, and sets a crawl delay based on what the file specifies. nod() moves the session to a new URL under the same host without repeating the robots check. scrape() runs the fetch:

library(polite)
library(rvest)

session <- bow("https://books.toscrape.com/", user_agent = "MyScraper (contact@example.com)")
page <- scrape(session)

titles <- page |>
  html_elements("article.product_pod h3 a") |>
  html_attr("title")

polite parses robots.txt, enforces the declared Crawl-delay (or a default 5-second delay if none is set), and caches responses so repeated calls to the same URL don’t re-fetch. nod() handles multi-page crawls on the same site:

pages <- 1:3 |>
  purrr::map(function(p) {
    session |>
      nod(sprintf("catalogue/page-%d.html", p)) |>
      scrape()
  })

Each nod reads the polite session’s rate limit and delays before fetching. Three pages take at least three times the crawl delay to complete.

Before writing any scraper, open <target-site>/robots.txt and check the User-agent block that matches your bot, plus Disallow and Crawl-delay. polite reads and enforces these automatically, but knowing what the file says helps you decide whether the crawl is worth building. Also skim the site’s Terms of Service. Some sites forbid automated scraping in ToS even if robots.txt allows it.

User-Agent and proxies

Every polite session takes a user_agent string. Use a real identifier that includes contact info, ideally a URL or email. The current list of browser User-Agents is worth bookmarking if you want to rotate through real values.

For proxied requests, chain httr2::req_proxy():

library(httr2)

resp <- request("https://books.toscrape.com/") |>
  req_proxy("http://user:pass@proxy.example.com:8080") |>
  req_perform()

For rotation across a proxy pool, proxies-for-web-scraping covers setup patterns you can port to R.

Automating R scrapers

Scraping in R turns into a scheduled job through operating-system-specific packages or a CI runner. On Windows, taskscheduleR writes entries into Task Scheduler. On macOS or Linux, cronR installs crontab entries. For runs off your local machine, GitHub Actions gives you free R runners on a cron schedule without owning a server.

Local scheduling

library(taskscheduleR)

taskscheduler_create(
  taskname = "books_scrape",
  rscript = "C:/scripts/books_scraper.R",
  schedule = "DAILY",
  starttime = "09:00"
)

That writes an entry into Windows Task Scheduler that runs the R script daily at 9 AM. The script runs under the R installation where taskscheduleR was installed.

cronR follows the same pattern on macOS and Linux:

library(cronR)

cron_add(
  command = cron_rscript("/home/user/scripts/books_scraper.R"),
  frequency = "daily",
  at = "09:00",
  id = "books_scrape"
)

That installs a crontab entry running the script daily at 9 AM.

GitHub Actions

For runs that don’t depend on a local machine staying online, GitHub Actions can install R, cache packages, and execute the script on a scheduled trigger. The r-lib/actions collection provides the official R setup steps:

# .github/workflows/scrape.yml
name: Daily scrape
on:
  schedule:
    - cron: "0 9 * * *"  # 09:00 UTC daily
  workflow_dispatch:

jobs:
  scrape:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: r-lib/actions/setup-r@v2
      - uses: r-lib/actions/setup-r-dependencies@v2
        with:
          packages: |
            rvest
            httr2
            readr
      - run: Rscript scripts/books_scraper.R
      - uses: actions/upload-artifact@v4
        with:
          name: scraped-books
          path: books.csv

The workflow runs on Ubuntu, installs R and the packages the script needs, executes the scraper, then uploads the CSV output as a workflow artifact.

Error handling

Any scheduled scraper eventually hits a failed page. Wrap the fetch in tryCatch so a single 500 or network hiccup doesn’t kill the whole run:

library(rvest)

safe_scrape <- function(url) {
  tryCatch({
    read_html(url) |>
      html_elements("article.product_pod h3 a") |>
      html_attr("title")
  },
  error = function(e) {
    message(sprintf("Failed to scrape %s: %s", url, conditionMessage(e)))
    character(0)
  })
}

The function returns book titles on success and an empty character vector on failure, with the error logged to stderr. Loops over safe_scrape(url) keep running even when a single URL fails.

For repeated failures on the same URL, add retry logic with httr2::req_retry() at the request level. That handles transient 5xx and 429 errors automatically before your tryCatch sees them.

FAQ

Is R good for web scraping? Yes for pipelines under a few hundred pages where the extracted data goes into analysis. rvest produces tibbles that plug directly into dplyr and ggplot2, saving the marshalling step you get in Python or Node. For crawls at thousands of pages per hour, Python or Node scale better because concurrency support in R is thin and chromote adds a ~5-second cold-start on each fresh script.

Is R or Python better for web scraping? Different strengths. R wins on tidy data pipelines and analytical downstream. Python wins on concurrency, ecosystem breadth (Scrapy, Playwright, BeautifulSoup), and community size. Use R when the scraper is one step in a data-analysis workflow. Python fits better when the scraper is the whole workflow.

What is rvest? The canonical scraping library in R. Wraps the xml2 package (which binds to libxml2 in C) with a tidyverse-flavored API for fetching HTML, selecting elements, and extracting text or attributes.

Is rvest part of Tidyverse? Yes. Maintained by Posit (formerly RStudio) and integrates with dplyr, purrr, tibble, and the rest of the tidyverse conventions.

Does rvest handle JavaScript? Not with read_html. For JavaScript-rendered pages, read_html_live() (introduced in rvest 1.0.4, February 2024) drives a real Chrome instance through chromote. RSelenium still works but requires Java and is being phased out for new projects.

Which R packages should I use? rvest for HTML parsing, httr2 for HTTP and hidden APIs, chromote for JavaScript-rendered pages, and polite for rate limits and robots.txt respect. RSelenium and Rcrawler still work on existing codebases.

Conclusion

The modern R scraping stack is rvest for static HTML, httr2 for hidden APIs, chromote (or read_html_live) for JavaScript-rendered pages, and polite for rate limits and robots.txt respect. Prefer XPath over CSS in rvest when extracting from small pages. Wrap scheduled scrapes in tryCatch and trigger them through cronR, taskscheduleR, or GitHub Actions.

For crawls above a few thousand pages per day, Python’s Scrapy or Node’s Crawlee scale better on concurrency. Everything else stays in R.