Web Scraping with Ruby in 2026

HTTParty and Nokogiri handle the majority of Ruby scraping work. HTTParty fetches the page and Nokogiri parses the HTML, letting you extract fields with CSS selectors or XPath. Login-protected pages need Mechanize, which submits forms and maintains session cookies without launching a browser. For pages that render content in JavaScript, Ferrum controls Chrome over the DevTools Protocol with no ChromeDriver binary to keep synchronized.

The HasData Web Scraping API fetches pages and executes JavaScript server-side, returning the final HTML.

Ruby gems for web scraping

The choice of gem depends on whether the page is static, needs a login flow, or renders content in JavaScript.

Most projects need only HTTParty, Nokogiri, and one browser automation gem.

Scraping static pages with HTTParty and Nokogiri

HTTParty.get returns a response object whose .body is a raw HTML string. Nokogiri::HTML turns that string into a traversable tree.

Add both to your Gemfile.

gem 'httparty'
gem 'nokogiri'

The example below fetches all book titles and prices from books.toscrape.com.

require 'httparty'
require 'nokogiri'

response = HTTParty.get('https://books.toscrape.com/')
doc = Nokogiri::HTML(response.body)

doc.css('article.product_pod').each do |book|
  title = book.css('h3 a').attr('title').value
  price = book.css('p.price_color').text.strip
  puts "#{title}: #{price}"
end

doc.css accepts any CSS selector and returns a NodeSet. Call .text on a node for its inner text, .attr('name') for an attribute value. doc.xpath takes XPath expressions and returns the same NodeSet type.

Scraping multiple pages

Most sites split listings across multiple pages. The loop follows the next-page link on each response and stops when none exists.

The example below collects all 1,000 books across 50 pages of books.toscrape.com.

require 'httparty'
require 'nokogiri'

base_url = 'https://books.toscrape.com/catalogue/'
next_path = 'page-1.html'
books = []

while next_path
  response = HTTParty.get(base_url + next_path)
  doc = Nokogiri::HTML(response.body)

  doc.css('article.product_pod').each do |book|
    books << {
      title: book.css('h3 a').attr('title').value,
      price: book.css('p.price_color').text.strip
    }
  end

  next_link = doc.css('li.next a').first
  next_path = next_link ? next_link.attr('href') : nil
  sleep(0.5) if next_path
end

puts "Scraped #{books.count} books"

The sleep(0.5) adds roughly 25 seconds to a full 50-page run, a small cost to stay below the thresholds where servers start returning 429s.

Scraping login-protected pages with Mechanize

Mechanize builds on Nokogiri and adds a cookie jar, redirect following, and form submission. It behaves like a browser that ignores JavaScript.

Add it to your Gemfile.

gem 'mechanize'

quotes.toscrape.com has a login form at /login. The example below signs in and scrapes the quote list behind it.

require 'mechanize'

agent = Mechanize.new
page = agent.get('http://quotes.toscrape.com/login')

form = page.form_with(action: '/login')
form.username = 'user'
form.password = 'password'
page = agent.submit(form)

page.search('.quote .text').each do |quote|
  puts quote.text
end

The agent object holds the session cookie automatically. Every agent.get call after submit sends that cookie along, so the server treats subsequent requests as authenticated. form_with(action: '/login') targets the login form specifically. On pages with multiple forms (search bars, newsletter inputs), page.forms.first would pick up the wrong one.

Scraping JavaScript-rendered pages with Ferrum

Ferrum connects to Chrome over the DevTools Protocol directly, skipping the ChromeDriver binary that Selenium depends on. That binary must match the installed Chrome version exactly, which breaks in CI environments and after browser auto-updates.

Add it to your Gemfile. Chrome or Chromium must be installed on the system.

gem 'ferrum'

quotes.toscrape.com has a JavaScript-rendered version at /js/ where the quote list is injected after page load. The example below scrapes it.

require 'ferrum'
require 'nokogiri'

browser = Ferrum::Browser.new(headless: true)
browser.go_to('https://quotes.toscrape.com/js/')
browser.network.wait_for_idle

doc = Nokogiri::HTML(browser.body)
doc.css('.quote .text').each do |quote|
  puts quote.text
end

browser.quit

network.wait_for_idle pauses until all pending network requests settle, which catches most content injected on page load. For pages that render on a timer rather than a fetch, a short sleep after go_to is more reliable.

Saving scraped data

Ruby’s standard library covers both CSV and JSON without extra gems.

To write a CSV file from the books array built earlier:

require 'csv'

CSV.open('books.csv', 'w', headers: ['title', 'price'], write_headers: true) do |csv|
  books.each { |book| csv << [book[:title], book[:price]] }
end

For JSON:

require 'json'

File.write('books.json', JSON.pretty_generate(books))

JSON.pretty_generate adds indentation, which makes the output readable without a viewer. For large datasets where file size matters, swap it for JSON.generate.

Using the HasData Web Scraping API

The HasData Web Scraping API takes a URL and returns rendered page content. JavaScript execution and proxy routing happen server-side.

The API accepts a POST request and returns the page as HTML, markdown, or structured JSON. Any Ruby HTTP library can make the call.

require 'httparty'
require 'nokogiri'

response = HTTParty.post(
  'https://api.hasdata.com/scrape/web',
  headers: {
    'Content-Type' => 'application/json',
    'x-api-key' => ENV['HASDATA_API_KEY']
  },
  body: {
    url: 'https://quotes.toscrape.com/js/',
    outputFormat: ['html']
  }.to_json
)

doc = Nokogiri::HTML(response.body)
doc.css('.quote .text').each do |quote|
  puts quote.text
end

Set HASDATA_API_KEY as an environment variable rather than hardcoding it in the script. The outputFormat field accepts 'html', 'markdown', or 'json'. 'markdown' skips the Nokogiri parsing step and feeds directly into LLM pipelines.

Before reaching for Ferrum, check the page source. If the data appears in view-source:, HTTParty handles it without a browser. If the data is absent from the source but visible in the rendered page, JavaScript is inserting it and Ferrum or the HasData API is the shorter path.