How to Scrape Dynamic Content in Python

Scraping dynamic content in Python rarely needs a browser. I checked 51 pages from shops, news sites, documentation, developer platforms, market data, and job boards with one headless Chrome visit and one plain HTTP request each. Of the 37 that answered the plain request, 25 had the content in the HTML already, 8 carried it as JSON inside a script tag, 3 loaded it from a JSON endpoint that answered requests directly, and 1 opened its endpoint once Origin and Referer headers were set. Not one of them needed a browser for its main content. The 10 pages that refused the plain request refused headless Chrome as well, so the browser did not buy anything there either.

Time and memory point the same way. Fetching the same 100 quotes took 3.2 seconds through the page’s JSON endpoint and 9.3 seconds through ten JavaScript-rendered pages in Chrome, and the Python process stayed under 70 MB while Chrome’s processes peaked near 600 MB. The sections below follow the order of cost, from reading JSON out of the page source, through the Network tab and the endpoint behind it, to Playwright and Selenium for the pages that need a real browser, and end with the survey and the timings in full.

How to tell what a dynamic page needs

The check takes two minutes and decides whether the scraper is a 15-line requests script or a browser you need to keep alive. Open the page, press Ctrl+U (Cmd+Option+U on a Mac) for the source, and search for a value you can see on the screen, a price or a headline. Then open DevTools, switch to the Network tab, tick the Fetch/XHR filter, and reload.

The table reads top to bottom on purpose. Each row down costs more in code, CPU, and fragility, and most pages stop at the second or third row. The rest of this article follows the same order.

The data may already be in the HTML

Server-side rendering came back with Next.js, Nuxt, Remix, and Astro, and it brought a habit with it. The framework renders the page on the server and then embeds the same data as JSON in the HTML so the client-side code can take over without a second fetch. Next.js pages built on the pages router put it in <script id="__NEXT_DATA__">. The app router streams it as self.__next_f.push(...) chunks. Nuxt 2 uses window.__NUXT__ and Nuxt 3 a __NUXT_DATA__ script block, older Redux apps window.__PRELOADED_STATE__, and almost every shop and news site adds application/ld+json blocks for search engines, with prices, ratings, authors, and dates already structured.

All of that is readable without JavaScript. A probe that fetches the page once and reports what it finds saves you from opening a browser for nothing:

import json
import re
import sys

import requests
from bs4 import BeautifulSoup

STATE_VARS = re.compile(
    r"window\.(__INITIAL_STATE__|__NUXT__|__APOLLO_STATE__|__PRELOADED_STATE__|__remixContext)\s*="
)


def probe(url):
    html = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=20).text
    soup = BeautifulSoup(html, "lxml")
    next_data = soup.find("script", id="__NEXT_DATA__")
    ld_json = soup.find_all("script", type="application/ld+json")
    json_scripts = soup.find_all("script", type="application/json")
    state = STATE_VARS.search(html)
    return {
        "html_kb": len(html) // 1024,
        "__NEXT_DATA__ kb": len(next_data.string) // 1024 if next_data else 0,
        "next_flight_chunks": html.count("self.__next_f.push"),
        "ld+json blocks": len(ld_json),
        "application/json scripts": len(json_scripts),
        "state variable": state.group(1) if state else None,
    }


if __name__ == "__main__":
    for url in sys.argv[1:]:
        print(url)
        print(json.dumps(probe(url), indent=2))

Run against a crypto price tracker and a documentation site, the probe found a 542 KB __NEXT_DATA__ blob in an 866 KB page (the top-100 listing with prices, volumes, and market caps) and a 30 KB one on react.dev. The price tracker’s visible table is rendered by React, and the naive approach is to wait for it in a browser, while json.loads on that one script tag gives the same numbers without launching anything.

The pattern shows up in older markup too. The practice page quotes.toscrape.com/js renders its quotes with JavaScript, and its source contains under 100 characters of visible text against about 1,500 in the rendered DOM, so a requests + BeautifulSoup script sees an empty page. The quotes are still in the response, as a JavaScript array the render loop reads from:

import json
import re

import requests

html = requests.get("https://quotes.toscrape.com/js/", timeout=20).text

# The page renders its quotes from a JavaScript array embedded in the HTML.
match = re.search(r"var data = (\[.*?\]);\s*$", html, re.S | re.M)
quotes = json.loads(match.group(1))

print(len(quotes), "quotes in the inline array")
for q in quotes[:3]:
    print(f"- {q['author']['name']}: {q['text'][:60]}... tags={q['tags']}")

The array is valid JSON here, which is common but not guaranteed. A literal with unquoted keys or trailing commas needs a tolerant parser such as json5 or demjson3, a value assembled from several variables needs the endpoint approach from the next section, and once the blob is loaded the rest is ordinary JSON parsing in Python, key paths included. One caveat on application/ld+json as well. It carries what the site wants Google to index, often a name, a price, and a rating, and rarely the full list on the page, so treat it as a shortcut for a product page and not for a category listing.

Finding the request the page makes

A page that shows a spinner and then fills a table got that table from somewhere, and the somewhere is almost always an HTTP request you can make yourself. Chrome’s Network panel shows every one of them. Open DevTools (F12), pick the Network tab, click the Fetch/XHR filter so images and scripts disappear, and reload the page. Sort by size. The entry that carries the data is usually the largest JSON response, and clicking it opens a Preview tab with the parsed structure, a Headers tab with the exact URL and request headers, and a Payload tab with the query string or POST body.

For pages that load on scroll or on a button, clear the list with the clear button (the circle with a slash), perform the action once, and watch the single request that appears. Its URL carries the pagination parameter (page=2, offset=20, cursor=abc), which is the loop variable of your scraper. Right-click the entry and choose Copy as cURL when the request has more headers than you want to type. The DevTools Network reference documents the filters and the copy options, and our own DevTools tips for scraping show how to turn a copied request into code.

Calling the endpoint directly

The Oscars page on scrapethissite.com renders one table per year and fetches each year with GET /pages/ajax-javascript/?ajax=true&year=YYYY, visible in the Network tab the moment you click a year. The response is a JSON list of films with their nomination and award counts, so the scraper is a loop over years and no HTML parsing at all:

import csv

import requests

# The Oscars page loads each year's table with
# GET /pages/ajax-javascript/?ajax=true&year=YYYY (seen in the Network tab).
BASE = "https://www.scrapethissite.com/pages/ajax-javascript/"
session = requests.Session()
session.headers["User-Agent"] = "Mozilla/5.0"

rows = []
for year in range(2010, 2016):
    films = session.get(BASE, params={"ajax": "true", "year": year}, timeout=20).json()
    for film in films:
        rows.append({"year": year, "title": film["title"], "nominations": film["nominations"],
                     "awards": film["awards"], "best_picture": film.get("best_picture", False)})
    print(year, len(films), "films, best picture:", next(f["title"] for f in films if f.get("best_picture")))

with open("oscars.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)
print(len(rows), "rows written")

Six requests, 87 rows, and a CSV with typed columns, because the endpoint hands back integers where the rendered table shows strings. The same logic covers infinite scroll. The scrolling page on quotes.toscrape.com calls /api/quotes?page=N, and the JSON includes a has_next flag, so the loop ends when the API says it ends instead of when a scroll stops adding elements:

import requests

# The infinite-scroll page fetches /api/quotes?page=N as you scroll.
# The JSON carries its own pagination flag, so the loop ends when the API says so.
API = "https://quotes.toscrape.com/api/quotes"
session = requests.Session()

page, quotes = 1, []
while True:
    data = session.get(API, params={"page": page}, timeout=20).json()
    quotes.extend(data["quotes"])
    if not data["has_next"]:
        break
    page += 1

print(f"{len(quotes)} quotes from {page} requests")
print(quotes[0]["author"]["name"], "-", quotes[0]["text"][:50])

The run printed 100 quotes from 10 requests. The browser version of the same job is in the infinite scroll section below, and the timing section puts the two side by side.

When the endpoint needs headers, cookies, or tokens

Some endpoints answer a plain requests.get with 401 or 403 while the browser gets 200, and the difference is always somewhere in the request. Compare the Headers tab of the working request in DevTools with what your script sends. In the survey, a market-data site’s economic-calendar endpoint answered 403 to a bare request and 200 with 1.2 MB of JSON once Origin and Referer were set to the site’s own address, a check that exists to stop other websites from embedding the feed and that costs a scraper two header lines. X-Requested-With: XMLHttpRequest and Accept: application/json are the other two headers frameworks look for. Cookies come next. A requests.Session that loads the page first collects them the way the browser did, and a CSRF token printed into the page HTML or a <meta> tag goes into the header or form field the JavaScript would put it in. The practice site’s login form shows the pattern end to end:

import re

import requests

# The login form carries a CSRF token. The endpoint call then reuses the session.
session = requests.Session()
session.headers["User-Agent"] = "Mozilla/5.0"

login_page = session.get("https://quotes.toscrape.com/login", timeout=20).text
token = re.search(r'name="csrf_token" value="([^"]+)"', login_page).group(1)
session.post("https://quotes.toscrape.com/login", timeout=20,
             data={"csrf_token": token, "username": "reader", "password": "reader"})
print("cookies after login:", list(session.cookies.get_dict()))

# Headers the browser sends with the XHR and a plain requests call does not.
headers = {"Referer": "https://quotes.toscrape.com/scroll", "X-Requested-With": "XMLHttpRequest",
           "Accept": "application/json, text/plain, */*"}
data = session.get("https://quotes.toscrape.com/api/quotes", params={"page": 1}, headers=headers, timeout=20).json()
print(len(data["quotes"]), "quotes through the logged-in session")

The run printed cookies after login: ['session'] and 10 quotes through the logged-in session. Bearer tokens follow the same route one step earlier, since the token itself comes from a request you can see in the Network tab, usually a POST to an auth endpoint whose response you store and reuse. What does not replay is a signature computed in JavaScript from the request itself, a timestamp hash in the URL or an encrypted body. Reversing that code is possible and rarely worth the time, and that is the row of the table where a browser becomes the cheaper option.

When you do need a browser

Rows six to ten of the decision table are browser territory. Signed request URLs, content that appears after a click or a login flow you cannot replay, canvas and WebSocket data, and sites that refuse plain HTTP but accept a real browser all belong there. In the survey none of the 37 reachable pages fell into that group for their main content, and the 10 blocked ones blocked headless Chrome too. When a browser is the right tool, Playwright and Selenium are the maintained choices, and Pyppeteer needs a caveat.

Playwright

Playwright drives Chromium, Firefox, and WebKit through one API, installs its own browser builds with playwright install, and waits for elements by default. The page below renders its quotes two seconds after load, which is exactly the case where a plain fetch returns an empty container:

from playwright.sync_api import sync_playwright

# This page renders its quotes with JavaScript two seconds after load.
URL = "https://quotes.toscrape.com/js-delayed/?delay=2000"

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto(URL)
    # Block until the first quote exists in the DOM, then read all of them.
    page.wait_for_selector(".quote", timeout=15000)
    quotes = page.query_selector_all(".quote")
    for quote in quotes[:3]:
        text = quote.query_selector(".text").inner_text()
        author = quote.query_selector(".author").inner_text()
        print(f"{author}: {text[:60]}")
    print(len(quotes), "quotes rendered")
    browser.close()

page.wait_for_selector() blocks until one matching element exists and raises a TimeoutError after the given time, and query_selector_all() then returns every match. Playwright has no wait_for_selector_all(). A script that calls it fails with AttributeError before the browser opens, and the pair above, one wait followed by query_selector_all(), is the correct form. The locator API expresses the same idea as page.locator(".quote"), whose actions such as inner_text() wait on their own, while locator.all() returns whatever matches at that moment and belongs after page.locator(".quote").first.wait_for().

Selenium

Selenium talks to the browser through WebDriver, and Selenium Manager, which is part of Selenium 4, downloads the matching driver on its own, so webdriver.Chrome() works on a machine with Chrome and nothing else. Explicit waits replace the time.sleep(5) that older scripts used:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

URL = "https://quotes.toscrape.com/js-delayed/?delay=2000"

options = Options()
options.add_argument("--headless=new")  # the current headless mode, a full Chrome without a window
driver = webdriver.Chrome(options=options)
try:
    driver.get(URL)
    # Wait up to 15 s for the JavaScript to insert the quotes, no fixed sleep.
    WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".quote")))
    quotes = driver.find_elements(By.CSS_SELECTOR, ".quote")
    for quote in quotes[:3]:
        print(quote.find_element(By.CSS_SELECTOR, ".author").text, "-",
              quote.find_element(By.CSS_SELECTOR, ".text").text[:60])
    print(len(quotes), "quotes rendered")
finally:
    driver.quit()

WebDriverWait(...).until(...) polls every half second and returns as soon as the condition holds, so the script waits about two seconds on this page instead of a fixed five. The try/finally is there because a Selenium script that raises before driver.quit() leaves a Chrome process behind, and ten such runs leave ten. The Selenium waits documentation lists the other conditions, and visibility_of_element_located and element_to_be_clickable are the two you will need next for the forms and clicks the Selenium scraping tutorial walks through.

Pyppeteer, and why not to start with it

Pyppeteer is a Python port of Puppeteer and it was a reasonable choice a few years ago. Its last release, 2.0.0, is more than two years old and the repository has been quiet for about as long, while Playwright and Selenium release a new version every month or two (1.62 and 4.47 as of this update). Pyppeteer also downloads its own Chromium build, pinned to revision 1181205, a browser several years behind the one your target’s anti-bot vendor tests against. Existing scripts, including the ones in our Pyppeteer tutorial, keep running, and the API is close enough to Playwright’s that page.waitForSelector becomes page.wait_for_selector and page.querySelectorAll becomes page.query_selector_all in a mechanical port. For new code, Playwright gives you the same async model with a maintained browser, and the headless browser comparison goes through the differences in detail.

Running the browser headless

Headless removes the window and nothing else. Chrome’s --headless=new mode, introduced in Chrome 112, is the full browser with the same rendering and network stack as the one on your desktop, only without a display. The older --headless implementation was a separate stripped-down build. Chrome 132 removed it from the main binary, and it lives on as the standalone chrome-headless-shell download, so on a current Chrome the plain --headless flag already means the new mode, and --headless=new says it explicitly on every version since 112. Playwright’s launch(headless=True) is the default and uses Playwright’s own headless shell build unless you pass a channel, and p.chromium.launch(channel="chrome", headless=True) runs your installed Chrome instead when a site treats the shell differently.

--disable-gpu and --no-sandbox travel with headless mode in old snippets and neither is needed today. --disable-gpu was a workaround for early headless builds on Windows, and --no-sandbox is a container workaround that disables a security layer, so it belongs in a Dockerfile you control and not in a script you share. What does matter is the window size. Headless Chrome 151 reports a 780x580 window through Selenium, and responsive sites hide columns or switch to a mobile layout at that width, so options.add_argument("--window-size=1366,900") keeps the desktop layout. Playwright sets a 1280x720 viewport on every new page, so the Playwright scripts above did not need it.

Executing JavaScript in the page

Sometimes the fastest way to get structured data out of a rendered page is to ask the page for it. Both libraries run a JavaScript function in the page and hand the return value back as Python objects, so a map over the DOM replaces twenty query_selector calls, and a global variable the page keeps for its own rendering is one expression away:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/")
    page.wait_for_selector(".quote")
    # Run JavaScript inside the page and get a Python list back.
    quotes = page.evaluate("""
        () => Array.from(document.querySelectorAll('.quote')).map(q => ({
            text: q.querySelector('.text').textContent,
            author: q.querySelector('.author').textContent,
            tags: Array.from(q.querySelectorAll('.tag')).map(t => t.textContent),
        }))
    """)
    # The page keeps its source array in a global variable, so read it directly.
    raw = page.evaluate("() => window.data ? data.length : null")
    print(len(quotes), "quotes from the DOM,", raw, "objects in the page's data array")
    browser.close()

The output was 10 quotes from the DOM, 10 objects in the page's data array, and the list of dicts came back ready for pandas or json.dump. Selenium’s equivalent is driver.execute_script("return ...") with the same rule that the return value must be JSON-serializable. On a page that assembles its state from several requests, window.__STORE__ or whatever the framework calls it is often the one place that has everything.

Infinite scroll

Feeds that load on scroll are where the endpoint route saves the most work, because the browser has to scroll, wait, and check for new elements while requests just increments a page number. When the endpoint is not reachable, the browser loop is short. Scroll to the bottom, wait until the number of items grows, and stop when a scroll adds nothing within a few seconds:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/scroll")
    page.wait_for_selector(".quote")
    while True:
        count = len(page.query_selector_all(".quote"))
        page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        try:
            # Stop when a scroll adds nothing within 3 s: the feed is exhausted.
            page.wait_for_function(f"document.querySelectorAll('.quote').length > {count}", timeout=3000)
        except Exception:
            break
    print(len(page.query_selector_all(".quote")), "quotes after scrolling")
    browser.close()

The script stops at 100 quotes, the same 100 that /api/quotes returned in ten requests. The wait_for_function call is what makes the loop reliable. The classic version compares document.body.scrollHeight before and after the scroll, and it breaks on pages whose height changes for other reasons, an ad slot loading or a footer expanding. Counting the items you actually want is immune to that. Sites with “Load more” buttons, sticky headers that hide the bottom, and scroll containers that are not the document body have their own tricks, and the Selenium scrolling guide covers them one by one.

How often each path works

The claim in the first paragraph comes from 51 URLs I picked before running anything. Seven practice pages from toscrape.com and scrapethissite.com served as controls, then ten documentation sites built on React, Vue, Angular, Svelte, Astro, and plain Sphinx, nine shop and catalog pages, seven news front pages, eight developer platforms, six finance, weather, and market pages, and four job and property listings. Each URL got one visit from headless Chrome 151 through Playwright, with no stealth patches and a 1366x900 viewport, and one plain requests.get with the same headers the browser had sent for the document. robots.txt was checked first, which removed Target, Etsy, Reuters, and IMDb from the sample.

The browser visit recorded every JSON response the page fetched and the final innerText. The plain fetch was checked for tagged JSON (__NEXT_DATA__, application/json, application/ld+json, Next.js flight chunks, window.__STATE__-style variables) and for JSON-compatible literals inside ordinary <script> blocks. To decide whether the plain HTML contained what the browser displayed, I split the rendered text into five-word sequences and counted how many of them occur in the plain fetch’s text. Pages at 0.85 or above counted as server-rendered. The ones between 0.5 and 0.85 I read by hand. Where the missing text was interface text that JavaScript fills in (the version switcher on docs.python.org, facet counts in Steam’s search sidebar, relative timestamps on TechCrunch and The Verge) the page counted as server-rendered, and where it was the content itself (Allbirds’ product grid, the personalized packages on the New York Times front page) the page went down the list. Below that, the page went to the inline-JSON path if the HTML carried a JSON blob of 5 KB or more, to the endpoint path if the largest same-site JSON response of 2 KB or more replayed with requests and no cookies, and to the browser otherwise.

The endpoint column includes the market-data page whose calendar endpoint needed Origin and Referer. Eleven of the 25 server-rendered pages also carried an inline JSON blob, so the structured path was open on 19 of the 37 reachable pages even where HTML parsing would have worked. The blocked column is the same ten sites for both clients. Six of them showed Cloudflare’s “Just a moment” page and IKEA Cloudflare’s “Attention Required”, H&M “Access Denied”, Indeed and Zillow their own block pages, and all ten returned 403 to headless Chrome with default settings as well as to requests. That is the practical meaning of the last two rows in the decision table. A browser by itself does not get past a bot check, and the fix is the same set of headers, proxies, and fingerprints whichever client sends the request.

A few pages deserve a closer look because they are the shapes you will meet most often.

The sample has limits I want to state. These are landing and listing pages, one visit each on one day from a home connection. Product detail pages, search results behind a form, and anything behind a login were not tested, and the blocked count would differ from a datacenter IP. The shares would move with a different sample, the order of checks would not.

What each path costs in time and memory

The same 100 quotes, ten pages of ten, fetched four ways from quotes.toscrape.com, medians of three runs on a home connection with Chrome 151 and no stealth patches:

The browser paths start at 3.3 to 3.6 seconds before the first quote arrives, which is the cost of launching Chrome and loading the page once, and the infinite-scroll browser then adds pages at about 0.07 s each because the page’s own script prefetches the next chunk as soon as the previous one lands. Opening ten separate JS-rendered pages is the shape most scrapers have, and there each page costs 0.66 s of rendering against 0.30 s for the equivalent JSON request. The endpoint path also parallelizes trivially, and ten threads brought the total to one second, while a browser is sequential by nature unless you pay for ten browsers. Memory is the larger difference at scale. The Python process stayed under 70 MB through the HTTP runs, and Chrome’s processes peaked between 540 and 615 MB for a single tab, so a fleet of headless browsers needs machines to match.

When the browser itself gets blocked

The last row of the decision table is the one where a site serves a challenge page to headless Chrome as well, or bans the IP after a few hundred renders. Fixing that yourself means stealth patches, residential proxies, and a browser pool to maintain, and at that point a managed browser is usually cheaper than the engineering time. HasData’s Web Scraping API renders the page on its side, through its own proxy pool, and the request is the same requests.post you already have, with jsRendering switched on and waitFor pointed at the selector the data appears in:

import os

import requests
from bs4 import BeautifulSoup

response = requests.post(
    "https://api.hasdata.com/scrape/web",
    headers={"x-api-key": os.environ["HASDATA_API_KEY"], "Content-Type": "application/json"},
    json={
        "url": "https://quotes.toscrape.com/js-delayed/?delay=2000",
        "jsRendering": True,          # render in a real browser on HasData's side
        "waitFor": ".quote",          # hold the response until this selector exists
        "outputFormat": ["html"],     # send back the rendered HTML, not a JSON envelope
    },
    timeout=90,
)
response.raise_for_status()
soup = BeautifulSoup(response.text, "lxml")
quotes = soup.select(".quote")
print(len(quotes), "quotes in the rendered HTML")
print(quotes[0].select_one(".author").text, "-", quotes[0].select_one(".text").text[:60])

The call returned the ten rendered quotes, and the rest of the pipeline is the BeautifulSoup code from the static case. A rendered request on datacenter proxies costs 10 credits (1 without rendering), residential proxies cost 5 without rendering and 15 with it, only successful responses are billed, and the 1,000 free credits at sign-up cover 100 rendered pages of testing, one request at a time on the free plan. waitFor fails the request if the selector never appears, which is the behaviour you want, since an empty page recorded as a success costs more to find later than a failed request.

Conclusion

Two minutes with view-source and the Fetch/XHR filter decide which of the four paths you are on. If the value is in the source, requests and BeautifulSoup are the scraper. If it sits in a <script> tag as JSON, json.loads is the parser and the DOM never enters the picture. If the page fetches it, the same fetch from requests ran at 0.30 seconds per extra page against 0.66 for a rendered page in this test, and it parallelizes with a thread pool. Playwright and Selenium remain the tools for signed requests, interactions, and content that exists only after a click, and their cost is the 3 to 4 seconds of startup and about 600 MB of memory per browser. Where a site blocked the plain request it blocked the plain browser too, which moves the problem to headers, proxies, and fingerprints, or to a service that handles them.