The Complete Guide to Web Scraping with Selenium in Python

This article covers everything you need to scrape with Selenium in Python. No theory, just practical examples: from setting up the driver and browser options to managing sessions, handling dynamic content, and scaling scrapers. Tested on Selenium 4.48 (older versions may not work).

Setup and Management

Selenium includes Selenium Manager since 4.6, which automatically finds and installs drivers, so you no longer need to install ChromeDriver or GeckoDriver manually.

Install Selenium

Install the latest Selenium (4.48, at the time of writing):

pip install selenium

The install brings Selenium Manager with it, so no separate driver download follows.

Driver Initialization

Initialize the webdriver for your browser:

from selenium import webdriver
 
# Chrome
driver = webdriver.Chrome()
 
# Firefox
driver = webdriver.Firefox()
 
# Edge
driver = webdriver.Edge()
 
# Safari (only for MacOS)
driver = webdriver.Safari()
 
driver.quit()

The result:

To use a specific driver binary, download it and specify its path:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
 
service = Service("/path/to/chromedriver")
driver = webdriver.Chrome(service=service)

In that case, Selenium will use the driver you pointed to instead of the built-in one.

Context Manager for Auto-Quit

A context manager automatically calls driver.quit() when the with block ends, even if an error occurs:

from selenium import webdriver
 
with webdriver.Chrome() as driver:
    driver.get("https://example.com")
    print(driver.title)

When running multiple scrapers, this pattern prevents missed browser processes, forgotten driver.quit(), and memory leaks.

Browser Options

Control how Selenium launches the browser (headless, user-agent, resource blocking, and more) using the Options classes and Chrome DevTools Protocol (CDP).

Headless Mode

Since Chrome 109+ (2023), the recommended headless usage is:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
 
options = Options()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)

Headless mode is usually pitched as a CPU and RAM saver. The benchmark below measures what it actually buys.

Window Size and User-Agent

When using a WebDriver, set at least a realistic window size and User-Agent. A headless browser starts with a small default viewport, which some sites read as a bot signal.

When setting a custom size, keep it within the actual visible area (window.innerWidth/window.innerHeight) and avoid unrealistic dimensions (for example, 500×500 or 1000×2000).

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
 
options = Options()
options.add_argument("--window-size=1920,1020")
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36")
 
driver = webdriver.Chrome(options=options)
 
# Check headers
driver.get("https://httpbin.org/headers")
print(driver.page_source)
driver.quit()

We maintain a list of the latest User Agents on our blog.

Disable JS/Images (prefs/CDP)

--disable-images and --disable-javascript flags no longer work in modern versions of Chrome.

# These flags do NOT work in modern Chrome
options.add_argument("--disable-images")
options.add_argument("--disable-javascript")

Chrome preferences let you configure profile options before launch, for example, enabling or disabling JavaScript, images, or notifications. This feature works only in Chrome and Chromium.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
 
options = Options()
prefs = {
    "profile.managed_default_content_settings.images": 2,      # disable images
    "profile.managed_default_content_settings.javascript": 2,  # disable JS
}
options.add_experimental_option("prefs", prefs)
 
driver = webdriver.Chrome(options=options)

For Firefox, preferences work the same way using set_preference().

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
 
options = Options()
options.set_preference("permissions.default.image", 2)  # disable images
options.set_preference("javascript.enabled", False)      # disable JS
 
driver = webdriver.Firefox(options=options)

Selenium 4+ integrates with the Chrome DevTools Protocol (CDP), providing access to low-level browser features like resource blocking, network control, and more.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
 
options = Options()
driver = webdriver.Chrome(options=options)
 
# Enable Network domain for CDP
driver.execute_cdp_cmd("Network.enable", {})
 
# Block selected resource types
driver.execute_cdp_cmd("Network.setBlockedURLs", {"urls": ["*.png", "*.jpg", "*.css"]})

This functionality was deprecated in Firefox and fully removed in Selenium 4.29.

What Each Configuration Costs

The options above all promise savings, so we measured them. One scenario (open a page, wait for the data element) ran 20 times per configuration on Selenium 4.48 against a server-rendered table page weighing about 317 KB, medians reported. Memory is the RSS of the whole browser process tree after load, and traffic is the Resource Timing total.

Headless saves almost no memory (540 vs 547 MB), and its real win is the 19% faster page load. Disabling images through prefs did close to nothing here because the target page carries few images, so that switch only pays on image-heavy targets, while CDP blocking (which also catches CSS and fonts) trimmed 14% of traffic. Disabling JavaScript cut traffic by 81% and load time by a third, and it only works because this page is server-rendered. On a JS-rendered target the same switch returns an empty shell, so know which kind of page you are scraping before reaching for it.

The numbers are medians of 20 runs on one machine and one page, so treat the ratios as the finding, not the absolute milliseconds.

Navigation

All navigation actions work within the active browser session.

`get`, `refresh`, `back`, `forward`

When scraping, you’ll use four basic actions: open, refresh, go back, and go forward:

from selenium import webdriver
 
driver = webdriver.Chrome()
 
driver.get("https://example.com")  # Open a page
driver.refresh()                   # Reload
driver.back()                      # Go back
driver.forward()                   # Go forward

Example:

Page Properties (`title`, `current_url`, `page_source`)

To verify the current page, use current_url and title. page_source returns the full HTML.

from selenium import webdriver
 
driver = webdriver.Chrome()
 
driver.get("https://example.com")    
 
print(driver.title)              # Page title
print(driver.current_url)        # Current URL
print(driver.page_source)        # HTML snippet

Use page_source only when you need the full HTML, for example, to store it or pass it to an LLM. For targeted data extraction, use find_element() or find_elements().

Following Links (via `click` or `get`)

You can follow links by clicking an element or by navigating directly with get().

driver.get("https://example.com")    
 
# Click a link element
link = driver.find_element(By.LINK_TEXT, "More information...")
link.click()
 
# Follow link manually
url = link.get_attribute("href")
driver.get(url)

Use click() when you need to emulate a real user or get() isn’t possible.

Locating Elements

Selenium 4 uses the By locator API, and the older find_element_by_* methods were removed in 4.3. Import locators from selenium.webdriver.common.by:

from selenium import webdriver
from selenium.webdriver.common.by import By

These two imports cover every locator example below.

Locators (`id`, `name`, `css`, `xpath`)

Locate the element using any available method:

Basic locator examples:

# By Tag Name
element_tag = driver.find_element(By.TAG_NAME, "h1")
 
# By CSS Selector
element = driver.find_element(By.CSS_SELECTOR, "h1")
 
# By XPath
element = driver.find_element(By.XPATH, "//h1")

Also available: ID, NAME, CLASS_NAME, LINK_TEXT, PARTIAL_LINK_TEXT. The XPath side of Selenium with its axes and functions is a topic of its own.

`find_element` vs `find_elements`

find_element() returns one element – or throws an error if none found. If multiple elements match, find_element() returns the first one. find_elements() returns a list – or yields an empty list, if nothing matches.

# Single element (raises NoSuchElementException if not found)
button = driver.find_element(By.CSS_SELECTOR, "button.submit")
 
# Multiple elements (returns a list, empty if not found)
links = driver.find_elements(By.TAG_NAME, "a")
print(f"Found {len(links)} links")

An empty list instead of an exception is what makes find_elements the safe probe for optional blocks.

Using DevTools and Shadow DOM

Shadow DOM isolates a component’s internals from the global document. Standard querySelector and other locators often can’t access elements inside a shadow root.

<custom-card>
  #shadow-root
    <div class="title">You can't scrape me</div>
</custom-card>

For pages using Shadow DOM, use JavaScript to access shadow roots:

host = driver.find_element(By.CSS_SELECTOR, "custom-card")
shadow_root = driver.execute_script("return arguments[0].shadowRoot", host)
title = shadow_root.find_element(By.CSS_SELECTOR, ".title")

The returned shadow_root behaves like a regular search context, so the same find_element calls work inside it.

Data Extraction

Selenium extracts visible text and attributes from the DOM via WebElement.

`element.text` Normalization

.text returns the text visible on the page, so it’s not always clean, containing extra spaces, newlines, non-breaking spaces, or invisible characters. Clean it before saving (strip, normalize whitespace, replace NBSP, etc.):

element = driver.find_element("css selector", "h1.title")
text = element.text.strip()  # remove leading/trailing spaces
text = text.replace("\u00A0", " ")  # replace non-breaking spaces

Both cleanups earn their keep on real pages, where invisible whitespace survives rendering.

`get_attribute` (href, src, value)

.get_attribute() returns internal HTML attributes (such as href, src, or value), not visible text.

# Links
link = driver.find_element("css selector", "a.download")
href = link.get_attribute("href")
 
# Images
image = driver.find_element("css selector", "img#logo")
src = image.get_attribute("src")

get_attribute("src") returns the resolved absolute URL, not the raw attribute text.

Extract Lists and Tables

Use CSS selectors and .text to extract lists and tables.

# Extract a list
items = driver.find_elements("css selector", "ul#menu li")
menu = [item.text.strip() for item in items]
 
# Extract a table
rows = driver.find_elements("css selector", "table#data tr")
table_data = []
for row in rows:
    cols = row.find_elements("tag name", "td")
    table_data.append([col.text.strip() for col in cols])

The loop flattens the table into a plain list of lists, ready for serialization.

Export to CSV/JSON

Structure extracted data into variables that are easy to save. Then export to CSV or JSON.

import csv, json
 
# CSV
with open("data.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(["Name", "Price"])
    writer.writerows(table_data)
 
# JSON
with open("data.json", "w", encoding="utf-8") as f:
    json.dump(table_data, f, ensure_ascii=False, indent=2)

ensure_ascii=False keeps non-Latin characters readable in the file.

Waiting

Modern web pages often load elements dynamically. Interacting with them too early can cause NoSuchElementException or StaleElementReferenceException.

Implicit vs Explicit Waits

An implicit wait sets a global delay for all element searches.

driver.implicitly_wait(10)  # wait up to 10s

It will delay every find_element(s) call, even when unnecessary:

element = driver.find_element(By.ID, "login")

An explicit wait pauses execution until a specific condition is met. It’s more reliable than implicit waits.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
 
driver = webdriver.Chrome()
driver.get("https://example.com")
 
wait = WebDriverWait(driver, 10)
 
# Common conditions:
wait.until(EC.presence_of_element_located((By.ID, "submit")))  # in DOM via ID
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".item")))  # in DOM via CSS
 
wait.until(EC.visibility_of_element_located((By.ID, "submit")))   # visible via ID
wait.until(EC.visibility_of_element_located((By.NAME, "email")))   # visible via name
 
wait.until(EC.element_to_be_clickable((By.ID, "submit"))).click()   # clickable
wait.until(EC.text_to_be_present_in_element((By.TAG_NAME, "h1"), "Welcome")) # text present
wait.until(EC.url_contains("dashboard"))           # URL contains
 
driver.quit()

If you forget to call .until() when using explicit waits, no waiting will occur.

Handling Stale Elements

A StaleElementReferenceException occurs when an element is removed or replaced in the DOM – for example, after AJAX updates, partial re-renders, or frame switches. Re-locate the element whenever the DOM changes:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import StaleElementReferenceException
 
driver = webdriver.Chrome()
driver.get("https://example.com")
wait = WebDriverWait(driver, 10)
old_element = driver.find_element(By.ID, "submit")
 
try:
    # Wait until the old reference goes stale (DOM replaced),
    # then re-find the element and click the fresh reference
    wait.until(EC.staleness_of(old_element))
    driver.find_element(By.ID, "submit").click()
except StaleElementReferenceException:
    # Fallback — re-find once more if it went stale mid-click
    driver.find_element(By.ID, "submit").click()

The retry re-finds the element instead of reusing the old reference, which is the only cure for staleness.

Interactions

Below, you’ll find the common interactions you’ll need when building a scraper.

`click`, JS-click Fallback

Use the standard Selenium click, and keep the JS click as a fallback for stubborn elements.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
 
driver = webdriver.Chrome()
driver.get("https://example.com")
 
# Standard click
button = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.ID, "submit-btn"))
)
button.click()
 
# JS fallback
driver.execute_script("arguments[0].click();", button)

The JS click skips Selenium’s visibility checks, so keep it for elements the normal click cannot reach.

`send_keys`, Special Keys

Sometimes, it’s easier to send keys (like Enter) instead of clicking a button. Use keys to simplify the script when possible.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
 
driver = webdriver.Chrome()
driver.get("https://www.google.com/")
 
input_field = driver.find_element(By.CSS_SELECTOR, "textarea[name='q']")
input_field.send_keys("HasData Selenium complete guide")
input_field.send_keys(Keys.RETURN)  # press Enter

Keys.RETURN submits from inside the field and skips the button lookup.

Forms, Checkboxes, Selects

Form filling is common when login is required before scraping.

from selenium.webdriver.support.ui import Select
 
# Fill a form
driver.find_element(By.NAME, "username").send_keys("user123")
driver.find_element(By.NAME, "password").send_keys("secret")
 
# Submit form
driver.find_element(By.ID, "login-form").submit()
 
# Checkbox / radio
driver.find_element(By.ID, "agree").click()
 
# Select dropdown
dropdown = Select(driver.find_element(By.ID, "options"))
dropdown.select_by_visible_text("Option 2")

Auth is a big topic, see our dedicated article on handling authentication while scraping.

`execute_script` Snippets

Selenium lets custom JS execute on the page. Test the JS in the browser console before adding it to your script.

# Scroll an element into view
element = driver.find_element(By.ID, "bottom")
driver.execute_script("arguments[0].scrollIntoView();", element)
 
# Get computed style
color = driver.execute_script("return window.getComputedStyle(arguments[0]).color;", element)
print(color)

Computed style reflects what the user actually sees, including rules applied from stylesheets rather than inline attributes.

Scrolling and Dynamic Content

For pages that load content on scroll (e.g. Google Maps), scroll in Selenium before extracting data.

`scrollIntoView`, Custom JS Scroll

Select the scrollable element and scroll to the end:

from selenium.webdriver.common.by import By
 
element = driver.find_element(By.CSS_SELECTOR, "#scroll-area")
driver.execute_script("arguments[0].scrollIntoView({behavior: 'smooth', block: 'center'});", element)

Or scroll by a specific pixel value:

# Scroll down 500 pixels
driver.execute_script("window.scrollBy(0, 500);")

Horizontal scroll works the same way (e.g. (500, 0)).

Infinite Scroll/Load-More Patterns

For infinite scrolling, use a loop with a clear exit condition to prevent the script from hanging:

import time
 
last_height = driver.execute_script("return document.body.scrollHeight")
 
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(1)  # wait for new content to load
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break  # reached bottom
    last_height = new_height

The height comparison is the loop’s exit condition, so a page that keeps growing keeps the loop alive.

Scrolling Inside Elements and Iframes

To scroll inside an iframe, switch to it first, scroll, then switch back to the main window:

# Switch to iframe first
iframe = driver.find_element("css selector", "#iframe-id")
driver.switch_to.frame(iframe)
 
# Scroll inside a scrollable element
scrollable = driver.find_element("css selector", ".scrollable")
driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight;", scrollable)
 
# Switch back to main page
driver.switch_to.default_content()

Always switch back to the main window after completing your actions.

Tabs and Alerts

Sites often open new tabs (target="_blank"). If your script doesn’t switch tabs, it will keep operating on the previous one.

Open/Switch Tabs and Windows

Here’s how you switch, close, and list windows/tabs:

# Get all open tabs
tabs = driver.window_handles  # returns a list of window handles
 
# Switch to the new tab (usually the second one)
driver.switch_to.window(tabs[1])
 
# Close the new tab
driver.close()
 
# Switch back to the original tab
driver.switch_to.window(tabs[0])

Closing a tab doesn’t automatically switch back – call switch_to.window() to return to the previous tab.

Handle Alerts and Dialogs

Handle alerts and modal dialogs to bypass pop-ups that block access to content, such as newsletter subscriptions, terms of service, or age confirmations.

# Trigger an alert
driver.find_element("id", "alertButton").click()
 
# Switch to the alert
alert = driver.switch_to.alert
print(alert.text)  # show alert text
 
# Accept or dismiss
alert.accept()      # click OK
# alert.dismiss()   # click Cancel

Always handle alerts in scripts that may trigger pop-ups. Otherwise, Selenium will throw an UnexpectedAlertPresentException.

Sessions and Authentication

If a site requires login, automate the process. It becomes essential when scaling scrapers.

Automate Logins (Form + Submit)

Basic login example:

driver.find_element(By.ID, "username").send_keys("my_user")
driver.find_element(By.ID, "password").send_keys("my_password")
driver.find_element(By.ID, "login-button").click()

Sites with CSRF tokens, redirects, or two-step forms need the longer authentication walkthrough.

Cookies (Save, Load, Export)

Cookies support get, add, and delete operations, plus export and import for reuse.

import json
from selenium import webdriver
from selenium.webdriver.common.by import By
 
# --- Save cookies ---
driver = webdriver.Chrome()
driver.get("https://example.com")
# assume logged in already
cookies = driver.get_cookies()
with open("cookies.json", "w") as f:
    json.dump(cookies, f)
driver.quit()
 
# --- Load cookies ---
driver = webdriver.Chrome()
driver.get("https://example.com")  # must open domain first
for cookie in json.load(open("cookies.json")):
    driver.add_cookie(cookie)
driver.refresh()
driver.quit()

There are helper libraries that simplify saving and loading cookies:

import pickle
 
# Save cookies
cookies = driver.get_cookies()
pickle.dump(cookies, open("cookies.pkl", "wb"))
 
# Load cookies
for c in pickle.load(open("cookies.pkl", "rb")):
    driver.add_cookie(c)

Load cookies only after opening the same domain, or the browser rejects them.

Preserve Sessions across Runs

Save and restore sessions using cookies. This works across sessions for the same domain.

import json
from selenium import webdriver
 
driver = webdriver.Chrome()
driver.get("https://example.com")
# after login
json.dump(driver.get_cookies(), open("cookies.json", "w"))
driver.quit()
 
# restore cookies
driver = webdriver.Chrome()
driver.get("https://example.com")
for cookie in json.load(open("cookies.json")):
    driver.add_cookie(cookie)
driver.refresh()
driver.quit()

A browser profile and user-data directory preserve cookies, local storage, cache, and session data.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
 
options = Options()
options.add_argument("user-data-dir=./chrome-profile")  # custom Chrome profile
 
driver = webdriver.Chrome(options=options)
driver.get("https://example.com")
print(driver.current_url)  # session preserved
driver.quit()

Reattaching a new WebDriver object to a running session has no supported route in the Python bindings. The old trick of copying command_executor._url and overwriting session_id broke in Selenium 4 (the constructor lost desired_capabilities, and the executor no longer exposes _url), and even when it ran it spawned an orphan browser. Keep the driver process alive for the session’s lifetime, or persist the state itself, the cookies above or a browser profile via --user-data-dir, and start a fresh driver on it.

Local and session storages are useful for SPA apps (React/Vue/Angular) or token-based auth.

driver = webdriver.Chrome()
driver.get("https://example.com")
 
# Set a token in localStorage
driver.execute_script("localStorage.setItem('auth_token','12345')")
 
# Retrieve token
token = driver.execute_script("return localStorage.getItem('auth_token')")

The same execute_script route reads anything else the page keeps in localStorage or sessionStorage.

Files and Snapshots

To manage file downloads, configure browser preferences to define the download directory and disable prompts. Selenium can retrieve page HTML, screenshots, and image URLs, but downloading image files requires a separate HTTP client, such as Requests.

Trigger and Monitor Downloads

This method works for portals that deliver files through a “Download” button – for example, reports or statements:

from selenium import webdriver
 
options = webdriver.ChromeOptions()
options.add_experimental_option("prefs", {"download.default_directory": "/downloads"})
driver = webdriver.Chrome(options=options)
 
driver.get("https://example.com")
driver.find_element("id", "downloadButton").click()
 
driver.quit()

The click returns immediately, so give the download time to finish (or watch the folder) before quitting the driver.

Auto-Download Setup

Configure Selenium to download files without prompts – otherwise, a save dialog may block the script.

options = webdriver.ChromeOptions()
options.add_experimental_option("prefs", {
    "download.default_directory": "/downloads",  # folder for downloads
    "download.prompt_for_download": False, # don't ask
    "safebrowsing.enabled": True          # bypass safe-browsing warning
})
 
driver = webdriver.Chrome(options=options)

With these prefs Chrome saves into download.default_directory without showing the save dialog.

Save HTML/Screenshots

Save the page HTML and capture screenshots for debugging and archiving.

# Save HTML:
driver = webdriver.Chrome()
driver.get("https://example.com")
 
with open("page.html", "w", encoding="utf-8") as f:
    f.write(driver.page_source)
 
# Save Screenshot:
driver = webdriver.Chrome()
driver.get("https://example.com")
driver.save_screenshot("screenshot.png")

save_screenshot captures only the visible viewport.

Download Images by `src`

Find <img> elements and read their src attributes. To download the files, use an HTTP client such as requests.

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://example.com/")

images = driver.find_elements(By.TAG_NAME, "img")
for img in images:
    print(img.get_attribute("src"))

Some sites serve images via blob: or data: URIs. In that case, decode the Base64 payload or read the image responses from the performance log, the same pattern the debugging section below uses for JSON.

Debugging and Error Handling

Add screenshots, logs, and retries to stabilize scrapers.

Screenshots on Error

Don’t capture every page you scrape – save screenshots only when errors occur. It makes debugging faster.

try:
    element = driver.find_element("id", "nonexistent")
except Exception as e:
    driver.save_screenshot("error_screenshot.png")

A screenshot taken inside except preserves the exact page state that caused the failure.

Logging Actions

Log errors along with the page context where they occurred. Full-trace logging is optional.

import logging
from selenium import webdriver
 
# Basic logging setup
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
 
with webdriver.Chrome() as driver:
    logging.info("Opening website...")
    driver.get("https://example.com")
    logging.info("Website loaded. Current URL: %s", driver.current_url)

The %s placeholders keep formatting lazy, so disabled log levels cost nothing.

Common Exceptions and Retry Strategies

Some errors are transient. A reload and a short wait often helps. Here are some common exceptions:

Example:

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException, StaleElementReferenceException, TimeoutException
 
driver = webdriver.Chrome()
driver.get("https://example.com")
 
try:
    el = driver.find_element("id", "dynamic-element")
    print(el.text)
except NoSuchElementException:
    print("Element not found!")
except StaleElementReferenceException:
    print("Element became stale!")
except TimeoutException:
    print("Operation timed out!")
 
driver.quit()

Set a clear limit on retry attempts when implementing retries.

Network and Proxy

Sometimes, data is available via API endpoints or XHRs that are easier to parse. When scraping is required, network skills can help you make scrapers faster and more reliable.

Blocking Resources over CDP

Use CDP (execute_cdp_cmd) to block images, fonts, CSS, or URL patterns to reduce bandwidth and speed up scraping.

driver.execute_cdp_cmd("Network.enable", {})
driver.execute_cdp_cmd("Network.setBlockedURLs", {"urls": [
    "*.jpg", "*.png", "*.gif", "*.woff", "*.css"
]})

The patterns take effect for every navigation that follows, with no browser restart.

Inspect Requests Without Selenium Wire

selenium-wire used to be the standard tool for reading browser traffic, and it is dead. The repository has been archived since the start of 2024, and it did not survive our compatibility check on a current stack. A fresh install fails to import on a current stack (pkg_resources left setuptools 81, so the break is not tied to a Python version), pinning setuptools<81 moves the failure to blinker._saferef, and with blinker<1.6 pinned as well the import finally passes, only for its bundled mitmproxy to crash on today’s TLS, leaving Chrome with net::ERR_CONNECTION_CLOSED. Remove it from new projects.

Chrome’s performance log covers the same job, capturing XHR/JSON responses, with nothing extra to install:

import json
from selenium import webdriver

options = webdriver.ChromeOptions()
options.set_capability("goog:loggingPrefs", {"performance": "ALL"})
driver = webdriver.Chrome(options=options)
driver.get("https://example.com")

for entry in driver.get_log("performance"):
    event = json.loads(entry["message"])["message"]
    if event["method"] == "Network.responseReceived" and "/api/" in event["params"]["response"]["url"]:
        body = driver.execute_cdp_cmd("Network.getResponseBody",
                                      {"requestId": event["params"]["requestId"]})
        data = json.loads(body["body"])

The loop filters Network.responseReceived events by a URL substring and pulls each matching body over CDP, the same wait-for-XHR pattern selenium-wire used to provide.

Proxy Config (Auth/Rotation)

Browsers handle proxy authentication through UI prompts or built-in mechanisms. Injecting credentials in URLs or command-line flags is often ignored or blocked.

# attempt to include credentials in the proxy URL (frequently ignored)
options.add_argument("--proxy-server=http://user:pass@host:port")

selenium-wire used to close this gap with its proxy option, and with the project gone the working routes are a local forwarder that adds the credentials for you, or a small Chrome extension that answers the auth prompt.

On our blog, you can learn more about what proxies are, where to find free or paid proxy providers, how to use them with Python, and how to configure proxies in Selenium.

Bypassing Cloudflare and Bot Protections

Anti-bot defenses are among the most common challenges. There’s no universal fix – protections and bypasses evolve constantly.

The stock driver announces itself. On the bot.sannysoft.com detector board, plain headless Chrome failed 4 of the 25 checks in our run, while the same driver headed failed only 1, so part of what detectors catch is headless mode itself rather than WebDriver. The tools below exist to close that gap, and sites read the same signals to decide between the page, a challenge, and a block.

undetected-chromedriver is the most direct patch: a drop-in Chrome class that strips the automation markers before the first page loads. In the same run it brought the failed checks down to 3 in headless mode. The benchmark above shows what that costs. Cold start grows from 1.4 to 8.8 seconds because the tool patches the driver binary on first launch, and the browser tree takes about 60 MB more memory. On Python 3.12+ it also needs pip install setuptools next to it, since it still imports distutils.

import undetected_chromedriver as uc

driver = uc.Chrome(headless=True, use_subprocess=True)
driver.get("https://bot.sannysoft.com/")
driver.save_screenshot("check.png")
driver.quit()

selenium-stealth applies a similar set of JavaScript patches to a regular driver, and SeleniumBase bundles the same idea plus its own driver management into its UC mode:

pip install seleniumbase

Example:

from seleniumbase import SB
 
with SB(uc=True, headless=False) as sb:
    url = "https://httpbin.org/headers"
    sb.uc_open_with_reconnect(url, 3)
    html = sb.get_page_source()

SB(uc=True) manages its own patched driver, so it does not combine with an externally created webdriver.Chrome.

Scaling Strategies for Scrapers

Scale Selenium scrapers across multiple browsers or machines using Grid or Scrapy.

Selenium Grid/Multiple Instances

Use Selenium Grid to run scripts in parallel. It was built with Java but works with any client (including Python).

Download the Selenium server JAR and start a hub:

java -jar selenium-server-<version>.jar hub

By default, the hub listens at http://localhost:4444 and serves its console at /ui. Start a node:

java -jar selenium-server-<version>.jar node --hub http://localhost:4444

Configure Python to use the remote hub:

from selenium import webdriver

grid_url = "http://localhost:4444"
driver1 = webdriver.Remote(command_executor=grid_url, options=webdriver.ChromeOptions())
driver2 = webdriver.Remote(command_executor=grid_url, options=webdriver.ChromeOptions())

driver1.get("https://example.com")
driver2.get("https://hasdata.com")

Selenium 4 takes an options object here, since desired_capabilities was removed in 4.10. Grid itself was outside our test run, so these are the documented calls rather than measured ones.

driver1 and driver2 send jobs to the hub, which distributes them across nodes. If a node supports two browsers, both sessions run. If it only supports one, the second waits.

An alternative is Selenoid, a lighter and Docker-based solution, but support was discontinued last year.

Combine Selenium with Scrapy

The pattern of letting Scrapy manage scheduling while a browser handles rendering survives, but the package that carried it does not. scrapy-selenium 0.0.7, the latest release, still passes executable_path to the WebDriver constructor, a parameter Selenium 4 removed, so it cannot start a driver on any current Selenium. Treat the snippet below as the shape of the integration (a download middleware yielding SeleniumRequest), and implement it with a maintained middleware or your own:

pip install scrapy

Here’s how the integration looks in a Scrapy spider:

import scrapy
from scrapy_selenium import SeleniumRequest

class MySpider(scrapy.Spider):
    name = 'my_spider'

    def start_requests(self):
        yield SeleniumRequest(url='https://example.com', callback=self.parse)

    def parse(self, response):
        title = response.css("h1::text").get()
        yield {"title": title}

The spider still runs through Scrapy’s scheduler, so concurrency and retries stay on the Scrapy side.

Use HasData API

The easiest way to scale a scraper is to hand the browser work to a scraping API. HasData’s Web Scraping API renders JavaScript on its proxy pool and returns the page HTML or extracted JSON. You need an API key to use it.

Here’s an example of an async script that limits concurrency per account and uses an LLM to extract site names and emails:

import asyncio
import aiohttp
import requests
 
api_key = "YOUR_API_KEY"
 
# Get available concurrency for your API key
def get_available_concurrency():
    url = "https://api.hasdata.com/user/me/usage"
    headers = {
        "Content-Type": "application/json",
        "x-api-key": api_key
    }
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        data = response.json()
        # return available concurrency, fallback to 1
        return data.get("data", {}).get("availableConcurrency", 1)
    return 1
 
# Build payload for scraping a single URL
def make_payload(url):
    return {
        "url": url,
        "proxyType": "datacenter",
        "proxyCountry": "US",
        "jsRendering": True,
        "aiExtractRules": {
            "companyName": {"description": "Company name", "type": "string"},
            "email": {"description": "Email addresses", "type": "string"}
        }
    }
 
# Async function to scrape one site
async def scrape_site(session, url):
    payload = make_payload(url)
    async with session.post(
        "https://api.hasdata.com/scrape/web",
        headers={
            "x-api-key": api_key,
            "Content-Type": "application/json"
        },
        json=payload
    ) as response:
        if response.status == 200:
            data = await response.json()
            ai_resp = data.get("aiResponse", {})
 
            company = ai_resp.get("companyName", "-")
            emails = ai_resp.get("email", "")
 
            return {
                "url": url,
                "company": company,
                "emails": emails
            }
        else:
            return {"url": url, "error": response.status}
 
# Async scrape for multiple sites
async def scrape_all(urls):
    concurrency = get_available_concurrency()
    # limit concurrent connections to API's availableConcurrency
    connector = aiohttp.TCPConnector(limit=concurrency)
    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [scrape_site(session, url) for url in urls]
        results = await asyncio.gather(*tasks)
    return results
 
# Run scraper
if __name__ == "__main__":
    urls_to_scrape = [
        "https://example.com",
        "https://hasdata.com",
        # add more URLs
    ]
    all_results = asyncio.run(scrape_all(urls_to_scrape))
    for r in all_results:
        print(r)

The script reads the account’s available concurrency at runtime and sizes its connection pool to match, so the same code scales from a free key to a paid plan without edits. That is the whole ladder in one article: one local driver for one page, a Grid or a driver pool for a batch, Scrapy when scheduling matters, and an API when maintaining the browser fleet costs more than the data is worth.