How to Scrape WooCommerce Products with the Store API, Sitemap, or HTML

A WooCommerce store publishes its own catalogue as JSON. Unless the owner switches the REST API off, the store answers GET /wp-json/wc/store/v1/products with up to 100 products per request, no API key, no login, because that endpoint is what the storefront’s own cart and product blocks read. I found 808 WooCommerce stores through search and sent each one the same plain requests. 89% returned the catalogue through the Store API, 82% listed every product URL in a sitemap, and the shop page selectors that WooCommerce themes share worked on 74%. Plain requests were blocked on 6% of them.

So the order of work for a WooCommerce store is fixed before any parsing starts. Ask the Store API first. Read the product sitemap when you need every product URL or the API is switched off. Parse the HTML when neither is available, and send the requests through a rendering client with proxies when the store blocks plain ones. Each route below carries its numbers from the 808-store measurement and the request count for a catalogue of a given size.

Four ways to get a WooCommerce catalogue

The routes differ in what they return, in how many requests a catalogue costs, and in how often they were open on the stores I measured.

The Store API is where every attempt should start, and the measurement says it is where most attempts end.

The WooCommerce Store API returns the catalogue as JSON

The Store API is the unauthenticated half of WooCommerce’s two APIs. The other half, the REST API at /wp-json/wc/v3/, needs a consumer key and secret that only the store owner can generate, and it exposes orders and customers. The Store API exposes what a visitor can already see, products, categories, and the visitor’s own cart, which is why it needs no key. It returns only published products, and password-protected products come back with an empty description.

One request returns up to 100 products (per_page caps there), and the response headers X-WP-Total and X-WP-TotalPages tell you how many are left, so a catalogue of 1,000 products is 10 requests.

import csv

import requests

# The Store API is the endpoint the storefront itself reads. No key, no login, JSON out.
STORE = "https://seedsforgenerations.com"
ENDPOINT = f"{STORE}/wp-json/wc/store/v1/products"
HEADERS = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}

products = []
page = 1
while True:
    r = requests.get(ENDPOINT, params={"per_page": 100, "page": page}, headers=HEADERS, timeout=30)
    r.raise_for_status()
    batch = r.json()
    products.extend(batch)
    total_pages = int(r.headers.get("X-WP-TotalPages", 1))
    if page >= total_pages or not batch:
        break
    page += 1

with open("products.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.writer(f)
    w.writerow(["id", "name", "sku", "type", "price", "regular_price", "sale_price", "currency", "in_stock", "rating", "reviews", "categories", "image", "url"])
    for p in products:
        prices = p["prices"]
        w.writerow([
            p["id"], p["name"], p["sku"], p["type"], prices["price"], prices["regular_price"], prices["sale_price"],
            prices["currency_code"], p["is_in_stock"], p["average_rating"], p["review_count"],
            "|".join(c["name"] for c in p["categories"]), p["images"][0]["src"] if p["images"] else "", p["permalink"],
        ])

print(f"{len(products)} products in {page} request(s), total reported by the store: {r.headers.get('X-WP-Total')}")

Against seedsforgenerations.com, a seed store with 691 products, the script finished in 7 requests and printed 691 products in 7 request(s), total reported by the store: 691. One product from the response, trimmed to the fields most pipelines keep:

{
  "id": 113935,
  "name": "Wildflower Mix North American Shade",
  "slug": "wildflower-mix-north-american-shade",
  "permalink": "https://seedsforgenerations.com/product/wildflower-mix-north-american-shade/",
  "sku": "SGF603",
  "type": "simple",
  "on_sale": false,
  "is_in_stock": true,
  "average_rating": "0",
  "review_count": 0,
  "prices": {
    "price": "325",
    "regular_price": "325",
    "sale_price": "325",
    "currency_code": "USD",
    "currency_minor_unit": 2
  },
  "categories": [
    {"id": 133, "name": "Flower", "slug": "flower"},
    {"id": 415, "name": "Full Sun", "slug": "full-sun"}
  ],
  "images": [
    {
      "src": "https://seedsforgenerations.com/wp-content/uploads/sites/2/2026/07/North-American-Shade-Wildflower-Mix-Photo.jpg",
      "alt": "A variety of North American wildflowers in bloom..."
    }
  ]
}

The fields are the ones the storefront itself uses, and they need no cleaning. Prices arrive as strings in the currency’s minor unit ("325" is 3.25 USD, and currency_minor_unit says how many decimals to shift), stock is a boolean is_in_stock, and each product carries its permalink so the sitemap and HTML routes can be joined to it later.

Variation prices and stock are the one thing the collection endpoint leaves out. GET /products/{id} on a variation id returns them one at a time, or GET /products?type=variation&parent={id} returns every variation of one parent in a single call, each with its own sku, prices, and is_in_stock. The same list endpoint takes the filters the shop page uses:

GET /wp-json/wc/store/v1/products?search=tomato&per_page=100
GET /wp-json/wc/store/v1/products?category=flower&orderby=price&order=asc
GET /wp-json/wc/store/v1/products?on_sale=true
GET /wp-json/wc/store/v1/products?min_price=300&max_price=500          (minor units, so 3.00 to 5.00 USD)
GET /wp-json/wc/store/v1/products?stock_status[]=instock
GET /wp-json/wc/store/v1/products?type=variation&parent=99167          (every variation of one variable product)

Two things are worth planning around. The default listing is wider than the shop: it returns every published product, including the ones the store hides from its shop pages. On seedsforgenerations.com the API returns 691 products while the shop lists 675, and catalog_visibility=visible narrows the API to the same 675, so the 16 extras are unlisted bundles and specials that only a direct link reaches. A store can also change what the endpoint returns with plugins. On bloomscape.com the default listing has 118 products, catalog_visibility=search returns 172, and the product sitemap lists 184 URLs, every one of them a page that answers 200, so the sitemap is the check on whether the API gave you everything. And a store can switch the REST API off with a security plugin or a firewall rule, which is the 36 of 808 stores below where the endpoint answered 404 or rest_no_route. On those, the sitemap is the next stop.

What 808 WooCommerce stores actually expose

I found the stores through search rather than a curated list. WooCommerce shop pages share the sorting widget text (“Default sorting”, “Sort by popularity”, “Sort by latest”), so I queried Google and Bing for that phrase across 68 product categories, probed the 1,463 domains left after dropping the platform vendors and forums, and kept the 808 whose homepage loaded WooCommerce assets or whose Store API answered. Each store then got the same requests from one machine with a browser user agent, without proxies or rendering. The Store API endpoint, the product sitemap (Yoast’s product-sitemap.xml, then the sitemap index from robots.txt, then the WordPress core wp-sitemap.xml), the shop page, and one product page.

The Store API is the route to plan around. It answered on 720 of the 808 stores, and the median store has 81 published products, so for half of them the whole catalogue is one request. Only 57 stores had more than 1,000 products, the largest 10,046, which is 101 requests. The fields were as complete as the storefront needs them to be: all but 134 of the 46,543 products returned on first pages carried a price, every one a stock flag, 95% an image, 94% categories, 89% a description, and half a SKU. Ratings above zero appeared on 4% of products, so review data is not something to expect from this route. Where the API did not answer, the reasons split three ways: 36 stores returned 404 or rest_no_route (the REST API switched off by a security plugin), 30 blocked the request with a 403, 401, or 503 while the homepage loaded normally, and 22 answered with an empty list or a page of HTML. Another 332 of the candidate domains answered their homepage with a 403 and are not in the count, since nothing could be measured on them.

The sitemap and the API mostly agree, and where they disagree the sitemap is the longer list. On the 617 stores with both, the sitemap count was within 5% of the API total on 371, larger by more than 5% on 180, and smaller on 66. Yoast wrote 456 of the 660 sitemaps found, the WordPress core wp-sitemap.xml 159. The HTML route is the least uniform of the three. The li.product card matched on 594 stores, but inside the card the .woocommerce-Price-amount span matched on 412 of them, the a.woocommerce-LoopProduct-link on 344, and the .woocommerce-loop-product__title heading on 234, because block themes and page builders rename the title element while keeping the card. Product pages were steadier for the price (.woocommerce-Price-amount on 576 of 745 pages) than for the title (h1.product_title on 502), and 540 of the 745 pages carried a JSON-LD Product object with the name, SKU, price, and availability, which is worth reading before any CSS selector when it is there.

Scraping the shop pages with requests and BeautifulSoup

When the Store API is off, the shop page is the listing. WooCommerce renders every product card from the same template, so the card and price classes below hold across most themes: the measurement above found the li.product card on 594 of 808 stores, the price span on 412 of those, and the title heading class on 234, which is why the script below falls back to the card’s own heading and link text for the title. Open a shop page with DevTools to see them.

The script walks the pages until the “next” link disappears and writes one row per card. It needs requests, beautifulsoup4, and lxml, and Python 3.10 or later, the same setup as the Python scraping introduction.

import csv

import requests
from bs4 import BeautifulSoup

# Shop pages share one card markup: each product is an <li class="product">. Themes rename the title element
# more often than the price or the link, so the title falls back to the card's heading and then to the link text.
STORE = "https://seedsforgenerations.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}


def text(node, selector, attr=None):
    el = node.select_one(selector)
    if el is None:
        return ""
    return el.get(attr, "") if attr else el.get_text(" ", strip=True)


def image_url(img):
    # lazy-loading themes put a 1x1 SVG in src and the real file in data-src
    if img is None:
        return ""
    return img.get("data-src") or img.get("data-lazy-src") or img.get("src", "")


rows = []
page = 1
while True:
    url = f"{STORE}/shop/" if page == 1 else f"{STORE}/shop/page/{page}/"
    r = requests.get(url, headers=HEADERS, timeout=30)
    if r.status_code != 200:
        print(f"stopped at page {page}: HTTP {r.status_code}")
        break
    soup = BeautifulSoup(r.text, "lxml")
    cards = soup.select("li.product")
    if not cards:
        break
    for card in cards:
        rows.append({
            "title": text(card, ".woocommerce-loop-product__title") or text(card, "h2, h3") or text(card, "a.woocommerce-LoopProduct-link"),
            "price": text(card, ".price"),
            "rating": text(card, ".star-rating", "aria-label"),
            "image": image_url(card.select_one("img")),
            "url": text(card, "a.woocommerce-LoopProduct-link", "href"),
        })
    if not soup.select_one("a.next.page-numbers"):
        break
    page += 1

with open("shop_listing.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["title", "price", "rating", "image", "url"])
    w.writeheader()
    w.writerows(rows)

print(f"{len(rows)} products from {page} listing page(s)")

On seedsforgenerations.com it printed 675 products from 34 listing page(s), 16 fewer than the Store API’s 691 because the shop hides its unlisted bundles, and with a detail the API route never shows you: 641 of the 675 cards had a 1x1 SVG placeholder in src and the real image URL in data-src, which is why the script reads data-src first. A shop page showed 12 products on 122 of the 328 stores with a second page, 16 on 47, and anything from 9 to 45 elsewhere, so at the common size this route costs 84 requests for the 1,000 products the Store API delivers in 10, and each card carries less: no SKU, no stock count, no description, and the price is a display string with a currency symbol rather than a number.

The failure you will meet on this route is a 403 Forbidden from a store behind Cloudflare or a similar firewall. 47 of the 808 stores blocked at least one of the plain requests. A headless browser with Selenium or Playwright is one way past a JavaScript check, at the price of running a browser per page, and the next section is the shorter path.

When the store blocks plain requests

The Web Scraping API fetches the page through a proxy of a chosen type and country, renders it in a browser when asked, and returns the HTML. In the parser above only requests.get changes:

import os

import requests

API = "https://api.hasdata.com/scrape/web"
HEADERS = {"x-api-key": os.environ["HASDATA_API_KEY"], "Content-Type": "application/json"}


def fetch(url):
    body = {
        "url": url,
        "jsRendering": True,       # camelCase. An unknown key is ignored without an error.
        "proxyType": "datacenter",  # "residential" when datacenter IPs get a 403
        "proxyCountry": "US",       # prices and stock are shown for the exit country
        "outputFormat": ["html"],
    }
    r = requests.post(API, headers=HEADERS, json=body, timeout=120)
    r.raise_for_status()
    return r.text

The parameter names are camelCase, and this matters more than it looks. The API ignores a key it does not know without returning an error, so js_rendering and proxy_type in snake_case would leave the defaults in place, and a snake_case proxy_country leaves the exit country to the pool. On residential proxies the misspelled key leaves the exit country to the pool’s default, and a store that shows prices and stock per country hands back a different catalogue on the odd request without any error to catch. Datacenter proxies with rendering cost 10 credits a page and residential ones 15, plain fetches 1 and 5, and only successful requests are billed. The full parameter list is in the documentation.

The 47 stores that blocked a plain request got the same two requests again through the API. Datacenter proxies without rendering, the 1-credit configuration, got the shop page or the Store API back on 33 of them, which suggests most of these blocks key on the client rather than the network. Residential proxies with rendering recovered another 8, and 6 stores stayed closed either way. On seedsforgenerations.com, which blocks nothing, the rendered route returned the same 675 products from 34 pages as the plain one, at 10 credits a page (340 for the store) against the 7 requests the Store API needs for the whole catalogue, which is the case for trying the API first even when the rendering client is already set up.

Scraping product links and data from the sitemap

Most WordPress stores ship a product sitemap, and it is the only route that lists every published product URL in one or two requests with nothing to paginate and nothing hidden behind a “load more” button. Yoast SEO, the most common source, writes /product-sitemap.xml and splits large catalogues into product-sitemap2.xml and so on, listed in /sitemap_index.xml. WordPress itself has shipped a core sitemap since version 5.5 at /wp-sitemap.xml, with product URLs in wp-sitemap-posts-product-1.xml at up to 2,000 per file. Either way the store’s robots.txt names the index. In the measurement 660 of 808 stores had a readable product sitemap, 456 of them Yoast’s.

Fetching the sitemap and the product URLs

The product sitemap is a flat <urlset> of <url><loc> entries, and the first entry on many stores is the shop page itself, which the code below drops.

import re
import sqlite3
import time

import pandas as pd
import requests
from bs4 import BeautifulSoup

# The product sitemap lists every published product URL, so there is no pagination to walk and nothing to miss.
# Large catalogues are split across numbered files, which the loop below follows until the first missing one.
STORE = "https://seedsforgenerations.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}
LIMIT = 50  # product pages to visit in this run


def product_urls(store):
    # Yoast: /product-sitemap.xml, /product-sitemap2.xml, ... WordPress core: /wp-sitemap-posts-product-1.xml, -2.xml, ...
    for pattern in ("/product-sitemap{n}.xml", "/wp-sitemap-posts-product-{n}.xml"):
        urls = []
        for n in range(1, 100):
            suffix = "" if n == 1 and pattern.startswith("/product-sitemap") else str(n)
            r = requests.get(store + pattern.format(n=suffix), headers=HEADERS, timeout=30)
            if r.status_code != 200 or "<urlset" not in r.text:
                break
            urls += re.findall(r"<loc>([^<]+)</loc>", r.text)
        if urls:
            return [u for u in urls if u.rstrip("/") != store and not u.rstrip("/").endswith("/shop")]
    return []

The function returns the product URLs in sitemap order, and everything after this point is one request per URL.

Scraping the product pages and saving to CSV, JSON, or SQLite

A product page carries what the listing card does not, the SKU, the short description, the full gallery, and the rating with its count. The selectors come from WooCommerce’s single-product template and held on 576 of the 745 product pages for the price and 502 for the title, which is why the code below carries a fallback for the title.

def text(soup, selector, attr=None):
    el = soup.select_one(selector)
    if el is None:
        return ""
    return el.get(attr, "") if attr else el.get_text(" ", strip=True)


def image_url(img):
    if img is None:
        return ""
    return img.get("data-large_image") or img.get("data-src") or img.get("src", "")


urls = product_urls(STORE)
print(f"{len(urls)} product URLs in the sitemap")

rows = []
for url in urls[:LIMIT]:
    r = requests.get(url, headers=HEADERS, timeout=30)
    if r.status_code != 200:
        rows.append({"url": url, "status": r.status_code})
        continue
    soup = BeautifulSoup(r.text, "lxml")
    rows.append({
        "url": url,
        "status": 200,
        "title": text(soup, "h1.product_title") or text(soup, "h1"),  # page builders keep the h1 and drop the class
        "price": text(soup, ".summary .woocommerce-Price-amount") or text(soup, ".woocommerce-Price-amount"),
        "sku": text(soup, ".sku"),
        "rating": text(soup, ".summary .star-rating", "aria-label"),
        "short_description": text(soup, ".woocommerce-product-details__short-description"),
        "image": image_url(soup.select_one(".woocommerce-product-gallery img")),
    })
    time.sleep(0.5)

df = pd.DataFrame(rows)
df.to_csv("products_from_sitemap.csv", index=False)
df.to_json("products_from_sitemap.json", orient="records", force_ascii=False)
with sqlite3.connect("products.db") as conn:
    df.to_sql("products", conn, if_exists="replace", index=False)

ok = df[df["status"] == 200]
print(f"{len(ok)} product pages parsed, {ok['title'].astype(bool).sum()} with a title, {ok['price'].astype(bool).sum()} with a price")

On seedsforgenerations.com the run printed 50 product pages parsed, 50 with a title, 50 with a price. The sitemap listed 689 URLs to the API’s 691, and 12 of the 50 pages, ten bundles and two specials built with a page builder, keep the h1 but drop the product_title class, which is what the title fallback in the code is for. All 50 pages carried a SKU, 38 a short description, and one a rating. Pandas writes the same frame to CSV, JSON, and a SQLite table in three lines, which is why it replaces the csv module here. The cost is the request count. One product page per product means a 1,000-product store is 1,001 requests against the Store API’s 10. The sitemap also settles a question the API cannot. Where the two disagree, the sitemap is usually the longer list (on 180 of the 617 stores with both, against 66 the other way), so a pull that has to be complete starts from the sitemap and joins the API data onto it by permalink.

Which route to use

The request counts decide it. At the most common shop page size the three routes cost this much per catalogue:

Start with the Store API and treat the other two as fallbacks, because on 89% of stores the first request answers the whole question and the data is already typed. Use the sitemap when the API is off or when you need the full product pages anyway, for descriptions and galleries. Parse shop pages when a store has neither, which was 45 of 808 stores in the measurement. Put the Web Scraping API in front of whichever route the store blocks, with proxyCountry set to the country whose prices you want.

Conclusion

The WooCommerce catalogue question has a short answer on most stores. One request to /wp-json/wc/store/v1/products?per_page=100 and a loop over X-WP-TotalPages returns everything the store publishes. The sitemap gives the complete URL list when the API is closed, the shared theme markup makes the HTML route the same code for every store, and the parameter spelling on the API request decides which country’s prices you get back. The scripts above ran against seedsforgenerations.com as shown, and the same four requests answer the question for any other store in under a minute. The e-commerce scraping guide covers the other platforms the same way, and the URL discovery guide goes deeper on sitemaps and crawling when a store has neither an API nor a sitemap.