Web Scraping with Axios and Cheerio in Node.js

axios fetches HTML and Cheerio parses it with CSS selectors. Together they cover static page scraping without launching a browser. This guide is about that one pair, from npm install to a paginating scraper and the cases where it stops working. Choosing between Cheerio and the browser libraries is a different question, answered with measurements in the JavaScript scraping libraries comparison.

One disclosure before the install step. Node ships fetch built in since version 18, so fetching a page needs no dependency at all, and on the 50-page run at the end of this guide the two clients finish within 5% of each other. axios earns its place here for the scraper-shaped extras this guide leans on, a timeout option, an error object that carries the response, and interceptors for retries, or simply because a codebase already uses it and consistency beats churn. The axios against built-in fetch comparison weighs the two in detail, and the last section of this guide shows the one-line swap.

Static means the data is in the HTML the server sends back. Press Ctrl+U in Chrome on your target page and search for any text you see on screen. If it is there, axios and Cheerio will find it. If it only appears after JavaScript runs, Cheerio parses the empty shell and any selector returns nothing.

You need Node 22+ and basic async/await.

Setting up the project

Create a new folder, initialize a module-type package, and install both libraries.

mkdir cheerio-scraper && cd cheerio-scraper
npm init -y && npm pkg set type=module
npm install axios cheerio

npm pkg set type=module writes "type": "module" to package.json. Without it, import statements throw SyntaxError: Cannot use import statement outside a module and the project defaults to CommonJS.

Fetching HTML with axios

axios.get(url) returns a promise that resolves to a response object. The HTML string is at response.data.

import axios from 'axios';

const { data: html } = await axios.get('https://books.toscrape.com');
console.log(html.slice(0, 300));

On an open site that is enough. Most real targets reject the default axios/1.x.x User-Agent on contact. A browser-realistic header set fixes that.

import axios from 'axios';

const headers = {
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 15_7_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36',
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language': 'en-US,en;q=0.9',
  'Accept-Encoding': 'gzip, deflate, br',
};

const { data: html } = await axios.get('https://books.toscrape.com', { headers });

Copy the User-Agent string from Chrome’s DevTools Network tab, which shows exactly what a real browser sends on each request. Chrome ships a new major version every four to six weeks, so a version number from six months ago is a detection signal.

Timeouts

axios has a built-in timeout option in milliseconds, and rejects with code ECONNABORTED when a request exceeds it. Unlike the built-in fetch, which resolves on any HTTP response, axios throws on 4xx and 5xx status codes and sets error.response when the server replied with an error. When error.response is undefined, the request failed at the network level (timeout, DNS failure, or connection refused). One wrapper handles both cases:

async function fetchHtml(url) {
  try {
    const { data } = await axios.get(url, { headers, timeout: 10_000 });
    return data;
  } catch (error) {
    if (error.code === 'ECONNABORTED') {
      throw new Error(`Timeout after 10 s for ${url}`);
    }
    if (error.response) {
      throw new Error(`HTTP ${error.response.status} for ${url}`);
    }
    throw error;
  }
}

The wrapper turns the three failure shapes into one exception with the URL in it, which is what the retry loop below and any logging need.

Retries on transient failures

A 429 (rate-limited) or 503 (overloaded) usually clears on retry. A 404 is a real error. The URL is wrong and waiting changes nothing. Exponential backoff with a status check separates the two cases.

async function fetchWithRetry(url, retries = 3) {
  for (let attempt = 1; attempt <= retries; attempt++) {
    try {
      const { data } = await axios.get(url, { headers, timeout: 10_000 });
      return data;
    } catch (error) {
      const status = error.response?.status;
      if (attempt === retries || ![429, 503, 504].includes(status)) throw error;
      await new Promise(r => setTimeout(r, Math.min(1000 * 2 ** (attempt - 1), 8000)));
    }
  }
}

The delay starts at 1 second, doubles each retry, and caps at 8 seconds. Any status outside [429, 503, 504] (including 404 and network timeouts) throws immediately.

Parsing HTML with Cheerio

cheerio.load(html) returns a $ function that works like jQuery. .text() returns the joined text of an element and all its descendants, .attr('name') reads an attribute and returns undefined when it is absent, .map() iterates matching elements and .get() at the end turns the result into a plain array. Relative URLs need new URL(href, baseUrl).href to become absolute, and .trim() strips the whitespace HTML leaves around tag content.

import * as cheerio from 'cheerio';

const $ = cheerio.load(html);
console.log($('h1').text().trim()); // "All products"

const titles = $('article.product_pod h3 a')
  .map((_, el) => $(el).attr('title'))
  .get();
console.log(titles.slice(0, 3));
// ["A Light in the Attic", "Tipping the Velvet", "Soumission"]

const first = $('article.product_pod').first();
const priceText = first.find('.price_color').text().trim(); // "£51.77"
const href = first.find('h3 a').attr('href');               // "catalogue/a-light..."
const fullUrl = new URL(href, 'https://books.toscrape.com').href;
console.log(priceText, fullUrl);

Missing elements return an empty selection. $(el).find('.maybe').text() returns '' rather than throwing. Check .length to test whether an element exists.

CSS selector patterns

Cheerio supports the same selectors a browser does. Right-click any element in the DevTools Elements panel and choose Copy selector to get a working selector without guessing.

The CSS selectors cheat sheet covers :nth-child, :not, attribute operators, and other patterns that come up with real site markup.

Selectors that survive a redesign

A class like .price_color is a styling decision, and styling decisions change. Selectors that anchor on what the page needs for its own sake last longer. The text of a link the user clicks, the position of an element inside a block the layout depends on, and an attribute the page uses (alt, title, data-*) all qualify. When several selectors are plausible, try them in order and record which one fired, so a silent zero after a redesign shows up in the log as a changed selector rather than as an empty file.

import axios from 'axios';
import * as cheerio from 'cheerio';

const { data: html } = await axios.get('https://books.toscrape.com/', { timeout: 10_000 });
const $ = cheerio.load(html);

// 1. Anchor on text the site cannot rename without changing what users see
const nextHref = $('a').filter((_, el) => $(el).text().trim() === 'next').attr('href');

// 2. Anchor on structure: the price is the first paragraph inside the price block, whatever its class
const firstCard = $('article.product_pod').first();
const priceByPosition = firstCard.find('.product_price p').first().text().trim();

// 3. Anchor on an attribute the page needs for itself (alt, title, data-*) instead of a styling class
const titlesByAttr = firstCard.find('img[alt]').attr('alt');

// 4. Fall back through several selectors and report which one fired
const candidates = ['.product_price .price_color', '.price_color', 'p:contains("£")'];
const [selectorUsed, price] = candidates
  .map(sel => [sel, firstCard.find(sel).first().text().trim()])
  .find(([, text]) => text) ?? ['none', null];

console.log({ nextHref, priceByPosition, titlesByAttr, selectorUsed, price });

One run prints both price reads and which selector the fallback chain settled on.

{
  nextHref: 'catalogue/page-2.html',
  priceByPosition: '£51.77',
  titlesByAttr: 'A Light in the Attic',
  selectorUsed: '.product_price .price_color',
  price: '£51.77'
}

With the fallback list, a redesign that renames one class leaves the scraper running on the second selector, and the log shows which one stopped matching.

A complete scraper

Here is a full script that walks the books.toscrape.com catalog, normalizes the data, and writes JSON.

fetchHtml handles the request. extractBooks maps over each product card. It converts the price string to a number, the rating word to 1–5, and the relative link to an absolute URL. nextPageUrl reads the pagination link, and the while loop walks pages until it disappears.

// scraper.js
import axios from 'axios';
import * as cheerio from 'cheerio';
import { writeFileSync } from 'node:fs';

const BASE = 'https://books.toscrape.com/';
const RATINGS = { One: 1, Two: 2, Three: 3, Four: 4, Five: 5 };
const headers = {
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 15_7_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36',
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
};

async function fetchHtml(url) {
  const { data } = await axios.get(url, { headers, timeout: 10_000 });
  return data;
}

function extractBooks($) {
  return $('article.product_pod').map((_, el) => {
    const ratingWord = ($(el).find('.star-rating').attr('class') || '')
      .replace('star-rating ', '');
    const priceText = $(el).find('.price_color').text().trim();
    const href = $(el).find('h3 a').attr('href');
    return {
      title:  $(el).find('h3 a').attr('title'),
      price:  Number(priceText.replace(/[^\d.]/g, '')),
      rating: RATINGS[ratingWord] ?? null,
      url:    new URL(href, BASE).href,
    };
  }).get();
}

function nextPageUrl($, currentUrl) {
  const href = $('li.next a').attr('href');
  return href ? new URL(href, currentUrl).href : null;
}

const allBooks = [];
let url = BASE;

while (url) {
  const html = await fetchHtml(url);
  const $ = cheerio.load(html);
  allBooks.push(...extractBooks($));
  url = nextPageUrl($, url);
}

writeFileSync('books.json', JSON.stringify(allBooks, null, 2));
console.log(`Saved ${allBooks.length} books`);

Run with node scraper.js. I got all 1,000 books across 50 pages in about 16 seconds. Wrapping fetchHtml calls in p-limit(5) drops that to around 3.3 seconds by running five pages concurrently.

What the Same Job Costs With Each Client

To put numbers on the choices in this article, I ran the same 50-page walk four ways from one Node 22.18.0 process (axios 1.20.0, Cheerio 1.2.0, Playwright 1.62.1), with a 20 ms sampler recording peak RSS, and a Playwright run on the first 10 pages for comparison. One run each, on a home connection, so the numbers are indicative of proportions rather than of absolute speed.

The HTTP client barely matters once the page is on the wire. Sequential axios and fetch land within about 5% of each other. Concurrency is what changes the picture, five parallel requests cut the run from 15 seconds to 3, because the network round trips are the whole budget and parsing is a rounding error in it. Cheerio parses a 50 KB catalogue page in a few milliseconds, which is why the concurrent runs finish 1,000 books in the time Playwright needs for 200. Per page, Playwright took about 550 ms against 310 ms for a plain request, before counting the browser’s own startup.

The cold-start difference between the clients is real and small. Importing axios and Cheerio took 646 to 800 ms across seven fresh processes, Cheerio alone 448 to 459 ms, so axios adds roughly 200 ms to a script that already pays for Cheerio. A full cold run (imports, one GET, one parse) came out at a median of 1,176 ms with axios and 1,193 ms with fetch over ten processes each, because the request dominates. The Node.js fetch guide measured the imports on their own and got 109 ms against 469 ms, which is the same gap seen from the other side.

When Cheerio returns nothing

If a selector returns an empty collection on a page where the data is clearly visible in the browser, the page is JS-rendered. The HTML axios received contains the container but not the content. The page fills it from JavaScript after load. quotes.toscrape.com serves the same quotes both ways, which makes the difference easy to see:

import axios from 'axios';
import * as cheerio from 'cheerio';

// Same site, two versions of the same page: server-rendered and filled in by JavaScript.
for (const url of ['https://quotes.toscrape.com/', 'https://quotes.toscrape.com/js/']) {
  const { data: html } = await axios.get(url, { timeout: 10_000 });
  const $ = cheerio.load(html);
  console.log(url, '->', $('.quote').length, 'quotes in the HTML,', $('script').length, 'script tags,', html.length, 'bytes');
}

The static page and the JavaScript one answer differently on every count.

https://quotes.toscrape.com/ -> 10 quotes in the HTML, 0 script tags, 11021 bytes
https://quotes.toscrape.com/js/ -> 0 quotes in the HTML, 2 script tags, 5806 bytes

The /js/ page is half the size and carries two script tags, one of which holds the quotes as a JavaScript array. Cheerio sees the array as text inside <script>, and a regex or JSON.parse on that text recovers the data without a browser, which is the first thing to try.

Check the Network tab in DevTools, filtered to Fetch/XHR, and reload the page. Most dynamic sites call a JSON API on boot and render from that response. Hitting that endpoint with axios directly is about 10x faster than launching Playwright, since it skips browser startup and page render entirely. The Node.js scraping guide covers how to find those endpoints with DevTools.

When there is no JSON shortcut and a browser is required, the choice between Cheerio and a browser library is settled in the JavaScript scraping libraries comparison, which measures Cheerio, Puppeteer and Playwright on the same tasks. Anti-bot systems like Cloudflare and DataDome look at TLS fingerprints and JavaScript globals that axios with realistic headers does not have, and for those targets the Web Scraping API renders the page on its side, through datacenter or residential proxies, and returns the HTML for Cheerio to parse.

Ten Page Types and What Handles Them

The Ctrl+U test from the introduction sorts most pages, and the table covers the cases where the answer is less obvious. The last two columns are what to reach for and what to check first.

Two of the ten cases, the login wall and the challenge page, are decisions rather than code. The other eight are axios and Cheerio with a different first step, and the next section shows the three that come up most.

Cheerio Pitfalls on Real Pages

Books.toscrape.com is UTF-8, has no inline JSON and no scripts, and real pages have all three. The encoding case first, because it fails silently: axios decodes the body as UTF-8 whatever the server declared, so a windows-1251 or koi8-r page parses without an error and every non-ASCII character comes out as a replacement character. Ask for bytes and decode with the charset from the header:

import axios from 'axios';
import * as cheerio from 'cheerio';

// Ask axios for bytes, read the charset from the Content-Type header, decode yourself.
const res = await axios.get('http://lib.ru/', { responseType: 'arraybuffer', timeout: 10_000 });
const charset = /charset=([\w-]+)/i.exec(res.headers['content-type'])?.[1] ?? 'utf-8';
const html = new TextDecoder(charset).decode(res.data);
const $ = cheerio.load(html);

console.log(charset, '->', $('title').text().trim());
console.log('decoded as utf-8 instead:', cheerio.load(Buffer.from(res.data).toString('utf8'))('title').text().trim().slice(0, 40));

Decoded from the declared charset the title reads, and forced through UTF-8 it is mojibake.

koi8-r -> Lib.Ru: Библиотека Максима Мошкова
decoded as utf-8 instead: Lib.Ru: ���������� ������� �������

Inline JSON is the opposite case, a page that gives you more than the visible HTML. Publishers and shops describe the page in a schema.org block for Google, and frameworks like Next.js ship the state the page was rendered from. Both are <script> contents, so $(el).html() returns the raw text and JSON.parse does the rest:

import axios from 'axios';
import * as cheerio from 'cheerio';

const headers = { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 15_7_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36' };
const { data: html } = await axios.get('https://www.bicycling.com/bikes-gear/a22132137/best-electric-bikes/', { headers, timeout: 15_000 });
const $ = cheerio.load(html);

// 1. schema.org JSON-LD: the page describes itself in the head
const ld = $('script[type="application/ld+json"]').map((_, el) => JSON.parse($(el).html())).get().flat();
const article = ld.find(o => /Article/.test(o['@type']));
console.log(article?.['@type'], '|', article?.headline, '|', article?.datePublished);

// 2. Next.js state: the data the page was rendered from, before any client-side JavaScript ran
const next = JSON.parse($('#__NEXT_DATA__').html());
console.log('__NEXT_DATA__ keys:', Object.keys(next.props?.pageProps ?? next.props ?? {}).slice(0, 6));

Both sources come back populated on the first try.

NewsArticle | The Best Electric Bikes of 2026: 15 Editor-Recommended Picks From Sub-$1,000 City Rides to Trail-Ready E-MTBs | 2018-07-20T21:03:25.290104Z
__NEXT_DATA__ keys: [ 'data', 'lang_tag', 'displayType', 'layoutContextProps', 'alternates', 'template' ]

The JSON-LD gave the headline and the original publication date, which the visible page shows as “Updated” with a newer date, and the Next.js state carries the whole article as structured data under data. Both survive redesigns better than any CSS selector, because the schema block exists for search engines and the state block exists for the framework’s own hydration.

The third pitfall is .text() on a container that holds <script> or <style> tags. Cheerio returns their contents as text, so a word count or a keyword extraction on $('body').text() includes the JavaScript. Remove them first with $('script, style, noscript').remove() and collapse whitespace with .replace(/\s+/g, ' '), or select the content container (article, main) instead of body.

Using the Web Scraping API

The API accepts a URL and extraction rules and returns structured data regardless of whether the page is static or JS-rendered.

Sign up on HasData and copy your API key. This example scrapes article titles from medium.com.

import axios from 'axios';

const response = await axios.post(
  'https://api.hasdata.com/scrape/web',
  {
    url: 'https://medium.com/',
    extractRules: { Title: 'h2' },
  },
  {
    headers: {
      'x-api-key': process.env.HASDATA_API_KEY,
      'Content-Type': 'application/json',
    },
  }
);

console.log(response.data.extractedData.Title);

The structured data is at response.data.extractedData. Without extractRules the response carries the page HTML in content, ready for cheerio.load, so the same parsing code runs on pages axios could not fetch itself. jsRendering: true renders the page in a browser on the API side and proxyType: "residential" moves the request onto residential exits. The request costs 1 credit plain on datacenter proxies, 5 plain on residential, 10 rendered on datacenter and 15 rendered on residential, and the free account starts with 1,000. To find the right selector for extractRules, open DevTools on the target page, right-click the element, choose Inspect, then right-click the highlighted node and choose Copy selector.

The HasData dashboard also has a request builder that generates ready-made code for any URL you paste in.

The builder outputs ready-to-run code you can paste directly into a Node.js script.

fetch vs axios in 2026

Built-in fetch is available in Node 22 with no installation and saves the axios import, about 200 ms of cold start in the measurement above and 2.1 MB in node_modules. On the 50-page run the two clients finished within about 5% of each other, so for a new scraper the choice comes down to the API you prefer rather than to speed.

The Cheerio integration is identical either way. Swap axios.get() for fetch(), read the body as text, and add the timeout yourself with AbortSignal.timeout, because fetch has no timeout option:

import * as cheerio from 'cheerio';

async function fetchHtml(url) {
  const res = await fetch(url, { headers, signal: AbortSignal.timeout(10_000) });
  if (!res.ok) throw new Error(`HTTP ${res.status} for ${url}`);
  return res.text();
}

const $ = cheerio.load(await fetchHtml('https://books.toscrape.com/'));
console.log($('article.product_pod').length); // 20

fetch resolves on any HTTP response including 4xx and 5xx, so res.ok must be checked manually. axios throws on non-2xx responses by default, which is why the fetchWithRetry function above reads the status from error.response rather than the resolved value.

Use axios when the project already depends on it, when you need request interceptors or upload progress tracking, or when you want automatic JSON parsing. For scraping work that is just GET requests returning HTML, both clients work identically with Cheerio.