Headless Browser in Python

A headless browser loads pages and runs JavaScript like a normal browser, without drawing a window. Dropping the rendering saves memory and startup time, which is why scrapers and test suites run this way. A headless browser in Python today means one of three libraries. Playwright and Selenium are both maintained and cover every mainstream engine, and Pyppeteer still appears in older codebases. This guide installs each one, gets a first script running, then covers parallel pages, proxies, user agents, profiles, and the newer option of handing the browser to an AI agent.

Choosing a Headless Browser for Python

Playwright is the place to start. It downloads its own browser builds, waits for elements on its own, and offers the same API in sync and async flavors, so a script grows from a one-off check into a parallel scraper without changing libraries. The first script doesn’t have to be typed at all either, since playwright codegen records one. The other options each hold a narrower niche:

Selenium makes sense when a team already runs it for testing, since the grid, the wrappers, and years of Stack Overflow answers all transfer to scraping. Our Selenium scraping guide covers that route end to end. Pyppeteer, the Python port of Puppeteer, went dormant with its 2.0.0 release in February 2024, so I’d reach for it only to maintain scripts that already use it. MechanicalSoup doesn’t drive a browser at all. It submits forms over plain HTTP, which works on static pages and stops working the moment content arrives by JavaScript.

Your First Headless Script

Playwright installs in two commands. The second one downloads the browser builds, so there’s no driver to match against a local Chrome version:

pip install playwright
playwright install chromium

The examples use books.toscrape.com, a sandbox built for scraping practice. demo.opencart.com, the demo shop older tutorials reach for, now answers a headless browser with a Cloudflare “Just a moment…” interstitial instead of the page, so it’s useless as a first target.

A minimal script opens the browser, loads the page, and reads the title:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://books.toscrape.com/")
    print("Page title:", page.title())
    browser.close()

It prints:

Page title: All products | Books to Scrape - Sandbox

Playwright also writes that draft for you. playwright codegen https://books.toscrape.com/ opens the page beside an Inspector window, and every click, fill and navigation turns into Python as you perform it. Python is the default target, and --target python-async or --target python-pytest change the shape of what comes out.

The recorded code is verbose and its locators are whatever the recorder guessed, so it’s a draft rather than the script you keep. What it saves is the selector hunt. Click the element once in the recorder and read back which locator Playwright picked, which beats reading the DOM in DevTools on a page you don’t know yet.

The Selenium version needs one package, and since version 4.6 Selenium Manager downloads the matching chromedriver by itself, so the old ritual of pinning driver versions to browser versions is gone:

pip install selenium

The script mirrors the Playwright one, with the headless choice moved into Chrome options:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)
driver.get("https://books.toscrape.com/")
print("Page title:", driver.title)
driver.quit()

It prints the same title. --headless=new selects Chrome’s current headless implementation, which shares the codebase with the visible browser. The old --headless flag ran a separate stripped-down build that sites could tell apart more easily.

Screenshots

Headless launches still render every page to an off-screen buffer, so screenshots work with nothing on screen to photograph. Playwright captures past the fold in one call:

page.screenshot(path="books.png", full_page=True)

Selenium saves the visible viewport:

driver.save_screenshot("books.png")

The full-page shot of the books.toscrape catalogue comes out at 625 KB. At that cost, a screenshot per run is a cheap visual log for a scraper working unattended, and usually the first thing worth checking when it starts returning empty fields.

When a Plain Request Is Enough

A headless browser pays off only on pages that build their content with JavaScript, and there’s a two-minute way to check which kind is in front of you. The scraping sandbox has a JS-rendered variant of its quotes page, and a plain HTTP request against it comes back missing the data:

import requests
from bs4 import BeautifulSoup

r = requests.get("https://quotes.toscrape.com/js/")
soup = BeautifulSoup(r.text, "html.parser")
print(r.status_code, len(soup.select("div.quote")))

That prints:

200 0

The server answers 200 and the quotes still aren’t in the markup, because they arrive as a JavaScript variable the browser renders after load. The same URL through the headless script above counts 10 quote blocks. When the plain request already contains the data, requests with a parser beats any browser on speed and memory. When it comes back empty like this, the page needs rendering, and our dynamic content guide covers the middle path of finding the API the page itself calls.

Where Pyppeteer Stands

Pyppeteer scripts still run, with one setup change. The library tries to download its own Chromium on first launch, and on current systems that download fails, so the launch needs an explicit path to a browser build you already have. Any Chromium works, including the one Playwright installed:

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch(
        headless=True,
        executablePath=r"C:\path\to\chrome.exe",
    )
    page = await browser.newPage()
    await page.goto("https://books.toscrape.com/")
    print("Page title:", await page.title())
    await browser.close()

asyncio.run(main())

Pointed at a working build, this printed the same books.toscrape title as the other two. The API mirrors Puppeteer’s, so our Pyppeteer guide reads almost like Node documentation with await in Python syntax. For new projects, Playwright’s async API gives the same asynchronous style with a maintained library. The dependency pins are one more trap. pyppeteer 2.0.0 requires websockets >=10.0,<11.0 and urllib3 >=1.25.8,<2.0.0, so installing it drags both back to releases from 2022 and can break other libraries in the same environment.

Running Pages in Parallel

Scraping ten pages one after another spends most of its time waiting on the network. Playwright’s async API lets one script load several pages at once, and asyncio.gather collects the results:

import asyncio
from playwright.async_api import async_playwright

URLS = [
    "https://books.toscrape.com/",
    "https://quotes.toscrape.com/",
    "https://books.toscrape.com/catalogue/category/books/travel_2/index.html",
]

async def read_title(browser, url):
    page = await browser.new_page()
    await page.goto(url)
    title = await page.title()
    await page.close()
    return url, title

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        results = await asyncio.gather(*(read_title(browser, u) for u in URLS))
        await browser.close()
    for url, title in results:
        print(title, "<-", url)

asyncio.run(main())

All three pages share one browser process, so the cost of another URL is a tab, and all three titles come back in the time the slowest page takes. Selenium has no async API, so parallel work there means threads, one full browser per thread:

from concurrent.futures import ThreadPoolExecutor
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

URLS = ["https://books.toscrape.com/", "https://quotes.toscrape.com/"]

def read_title(url):
    options = Options()
    options.add_argument("--headless=new")
    driver = webdriver.Chrome(options=options)
    try:
        driver.get(url)
        return url, driver.title
    finally:
        driver.quit()

with ThreadPoolExecutor(max_workers=2) as pool:
    for url, title in pool.map(read_title, URLS):
        print(title, "<-", url)

Every worker here starts its own Chrome process, so the worker count is bounded by RAM rather than CPU. Two workers sat comfortably in memory here, each adding a few hundred megabytes. That difference, tabs against processes, is the practical reason the async route scales further on the same hardware.

Proxies, User Agents, and Profiles

Headless launches accept the same identity controls as visible ones.

Proxies

A proxy routes requests through another IP address, which spreads load across exits and keeps one address from accumulating the whole request history. Playwright takes the proxy as a launch option, credentials included:

browser = p.chromium.launch(
    headless=True,
    proxy={
        "server": "http://proxy_ip:proxy_port",
        "username": "user",
        "password": "pass",
    },
)

Selenium passes it as a Chrome argument:

options.add_argument("--proxy-server=http://proxy_ip:proxy_port")

Chrome’s flag takes only a host and port, so authenticated proxies in Selenium usually go through a local forwarder or a provider that whitelists your IP. Which proxies are worth routing through is its own topic, covered in our proxy basics guide and the overview of rotating proxy providers.

User Agents and Client Hints

Overriding the user agent is one line per library:

ua = "Mozilla/5.0 (Macintosh; Intel Mac OS X 15_8_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.0.0 Safari/537.36"

context = browser.new_context(user_agent=ua)   # Playwright
options.add_argument(f"user-agent={ua}")       # Selenium

The override reaches the server, and that’s also where it stops. Checking the headers a swapped Playwright context actually sends, the User-Agent line carried the mac Chrome string from the code, while the Sec-Ch-Ua client-hint header on the same request still read "HeadlessChrome";v="151". A server that reads client hints sees the real browser and the word Headless in it regardless of the string above. That gap is worth knowing before concluding that a user-agent swap made a script look like a person. Current strings to swap in are in our auto-updated user agent table.

Profiles

By default every launch starts a blank browser with no cookies and no history. Pointing the browser at a profile directory keeps state between runs, so a login survives and repeat visits look like repeat visits. Playwright does this with a persistent context, which replaces the separate launch and context objects:

context = p.chromium.launch_persistent_context(
    user_data_dir=r"C:\scraper-profile",
    headless=True,
)
page = context.new_page()

Selenium takes the directory as an argument:

options.add_argument(r"--user-data-dir=C:\scraper-profile")

Use a dedicated directory rather than your everyday browser profile. The browser locks the directory while it runs, and one crashed script inside a personal profile is a bad trade for saved cookies.

Driving a Headless Browser with an AI Agent

The newest way to run a headless browser in Python is to let a language model operate it. The browser-use library wires an LLM to a Playwright-managed Chromium, and its headless behavior is the detail people search for. The headless parameter defaults to None, which auto-detects the environment. With a display attached the browser opens visibly. On a server without one it runs headless. Passing headless=True forces it either way, and so does setting BROWSER_USE_HEADLESS in the environment, which the default reads before falling back to display detection. devtools=True only works with headless=False, since there’s no window to open the inspector in.

from browser_use import Agent, Browser, ChatOpenAI

browser = Browser(headless=True)
agent = Agent(
    task="Open https://books.toscrape.com/ and report the title of the first book",
    llm=ChatOpenAI(model="gpt-4.1-mini"),
    browser=browser,
)
agent.run_sync()

Starting this session boots a real Chromium in the background, which the imports and the headless=True launch set up. The agent loop itself needs an LLM API key, and each step costs model tokens, so the pattern fits tasks where the navigation is unpredictable enough that writing selectors by hand would be worse. For a fixed, known page structure, the plain Playwright script from the first section does the same work with no model in the loop.

Conclusion and Takeaways

Playwright covers new headless work in Python, Selenium carries the projects already built on it, and Pyppeteer is in maintenance mode with a two-year-old release to prove it. The parts around the launch call decide how the script behaves at volume. Parallelism through tabs beats parallelism through processes, a user-agent swap changes less than the client hints reveal, and a profile directory decides whether state persists between runs.

Running your own headless fleet also means owning those parts. When the goal is the rendered HTML rather than the infrastructure, our Web Scraping API executes the JavaScript, routes the request through its proxy pool, and returns the page for parsing.