How to Set Up a Proxy with Selenium in Python

One argument sets a proxy in Selenium:

options.add_argument("--proxy-server=http://194.126.37.94:8080")

That line covers every proxy that does not ask for a password. The moment yours does, the usual advice stops working, and most of what is written about it no longer runs on a current Chrome. This guide sets the proxy, shows what happened when eight authentication methods were tried against Chrome 151, rotates a pool, and reads the errors Chrome returns when something is wrong.

Why Use Proxies with Selenium?

A proxy sits between the browser and the site. The request goes to the proxy, the proxy forwards it, and the response comes back the same way, so the site records the proxy’s address rather than yours.

Three consequences matter for scraping. A site that counts requests per address sees a pool as many visitors instead of one, which is why a crawl spread over a pool meets HTTP 429 later than the same crawl from one machine. Pages that vary by country come back in the version the exit country sees. And the address that gets blocked is the proxy’s, so the fix is a different proxy rather than a new office IP.

The trade is speed. Every request takes an extra hop, residential pools add more, and a browser already costs more per page than an HTTP client.

Which pool you point Selenium at changes the arithmetic. Datacenter addresses are the fastest and the cheapest, and they are also the ones a site recognizes first, because whole ranges are registered to hosting companies. Residential addresses come from consumer connections and are treated as ordinary visitors, at a few hundred milliseconds more per request and a price per gigabyte. Mobile addresses cost the most and are worth it only where the content is served to phones alone. The types of proxies go into what each is for.

One more decision comes before any of this. If the page you need arrives as HTML, an HTTP client with a proxy is an order of magnitude cheaper than a browser with a proxy, and proxies in requests covers that. Selenium earns its cost when the page builds itself with JavaScript, when a login or a multi-step flow has to happen in a real session, or when the site checks browser behaviour that an HTTP client cannot fake.

Prerequisites

Python 3 and Selenium 4:

pip install selenium

Selenium 4.6 and later ship Selenium Manager, which downloads a matching driver on first run. The chromedriver-by-hand step that older tutorials open with is no longer needed, and webdriver.Chrome() with no arguments works as long as Chrome is installed. If you are new to the library, Selenium scraping covers the rest of the API, and proxies in requests covers the same job without a browser.

Everything below was run on Selenium 4.47, Chrome 151 and Python 3.14, against a local proxy so the results do not depend on a vendor’s network. Chrome is the browser here because it is what most Selenium scraping runs on. Firefox takes the same proxy through FirefoxOptions().set_preference("network.proxy.*") rather than a command-line switch, and its authentication story differs, so the measured table below applies to Chrome.

Setting Up a Proxy in Selenium 4

Chrome takes the proxy as a command-line switch, and Selenium passes it through Options:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--proxy-server=http://194.126.37.94:8080")

driver = webdriver.Chrome(options=options)
driver.get("https://books.toscrape.com/")
print("title:", driver.title)
print("first book:", driver.find_element("css selector", "article.product_pod h3 a").get_attribute("title"))
driver.quit()

Against a working proxy that run prints:

title: All products | Books to Scrape - Sandbox
first book: A Light in the Attic

The scheme in front of the address selects the protocol, and one helper keeps the choice in one place:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options


def chrome_with_proxy(proxy_url, headless=True, bypass=None):
    """proxy_url is http://host:port for an HTTP proxy or socks5://host:port for SOCKS5."""
    options = Options()
    if headless:
        options.add_argument("--headless=new")
    options.add_argument(f"--proxy-server={proxy_url}")
    if bypass:                                   # comma-separated hosts that skip the proxy
        options.add_argument(f"--proxy-bypass-list={bypass}")
    return webdriver.Chrome(options=options)


driver = chrome_with_proxy("http://194.126.37.94:8080")      # HTTP and HTTPS traffic
# driver = chrome_with_proxy("socks5://194.126.37.94:1080")  # SOCKS5, DNS resolved by the proxy

Three details save an hour later. Chrome accepts one --proxy-server value, so a second call replaces the first rather than adding a protocol, and an HTTP proxy carries HTTPS traffic through a CONNECT tunnel rather than needing a separate entry. SOCKS5 differs in one way that matters for scraping, since the proxy resolves the hostname, so DNS goes out through the proxy as well. And the old webdriver.Proxy class with DesiredCapabilities was removed in Selenium 4, which is why examples built on it raise AttributeError on a current install.

Checking that the proxy is really in the path

A proxy that is configured but never used is the quietest failure in this article, because the scraper works and every request goes out from your own address. Local addresses skip the proxy by default, so a test against localhost proves nothing.

Two checks settle it. Point the browser at a page that reports the address it sees, which every proxy provider offers and several public endpoints do, and compare it with your own. Or read the proxy’s log, where each page load appears as a CONNECT line for HTTPS traffic:

127.0.0.1:50071 - CONNECT books.toscrape.com:443 - 6314 bytes

If the log stays empty while pages load, the browser is going direct.

Proxy Authentication in Selenium

A paid proxy usually comes as user:password@host:port, and this is where the topic gets its reputation. Chrome ignores credentials in --proxy-server, so the line that looks obvious fails:

# does not authenticate: Chrome drops the credentials
options.add_argument("--proxy-server=http://scraper:s3cret@127.0.0.1:8899")

To find out what does work, I ran a local proxy that demands Basic auth and drove Chrome through it eight ways, headed and headless. The target was a static page, so a failure is the proxy refusing rather than the site.

The extension rows deserve a note, because that recipe is the one most articles and repositories publish. A control run with a trivial extension that only renames the page title never ran either, headed or headless, so the extension was not at fault: Chrome 151 ignored --load-extension on this machine. Anything built on that mechanism goes with it, which is the likely reason the seleniumbase option returned nothing.

That leaves two routes that worked.

Ask the provider for IP whitelisting

Most paid providers let you authorize your server’s address instead of sending a password. The proxy then answers without a challenge and the plain one-liner is enough:

options.add_argument("--proxy-server=http://194.126.37.94:8080")

This is the shortest path and the one to take when the scraper runs from a fixed address.

Put a forwarder in front of the proxy

When the credentials cannot be avoided, run a small proxy of your own on localhost, point Chrome at it with no password, and let it add the Proxy-Authorization header on the way out. It is about forty lines with nothing but the standard library:

import asyncio
import base64
import threading

UPSTREAM_HOST, UPSTREAM_PORT = "proxy.example.com", 8899
CREDENTIALS = base64.b64encode(b"scraper:s3cret").decode()
LISTEN_PORT = 8894


async def handle(reader, writer):
    head = await reader.readuntil(b"\r\n\r\n")
    first = head.split(b"\r\n", 1)[0].decode("latin-1")
    up_r, up_w = await asyncio.open_connection(UPSTREAM_HOST, UPSTREAM_PORT)
    auth = f"Proxy-Authorization: Basic {CREDENTIALS}".encode()
    if first.upper().startswith("CONNECT"):          # HTTPS goes through a tunnel
        target = first.split()[1].encode()
        up_w.write(b"CONNECT " + target + b" HTTP/1.1\r\nHost: " + target + b"\r\n" + auth + b"\r\n\r\n")
    else:                                            # plain HTTP keeps its own headers
        lines = [l for l in head.split(b"\r\n") if not l.lower().startswith(b"proxy-authorization")]
        lines.insert(1, auth)
        up_w.write(b"\r\n".join(lines))
    await up_w.drain()

    async def pipe(r, w):
        try:
            while (chunk := await r.read(65536)):
                w.write(chunk)
                await w.drain()
        finally:
            w.close()

    await asyncio.gather(pipe(up_r, writer), pipe(reader, up_w))


def serve():
    async def main():
        server = await asyncio.start_server(handle, "127.0.0.1", LISTEN_PORT)
        async with server:
            await server.serve_forever()
    asyncio.run(main())


threading.Thread(target=serve, daemon=True).start()

Chrome then needs no credentials at all, because the forwarder supplies them:

options = Options()
options.add_argument(f"--proxy-server=http://127.0.0.1:{LISTEN_PORT}")
driver = webdriver.Chrome(options=options)
driver.get("https://books.toscrape.com/")
print("through the forwarder:", driver.title)
driver.quit()

The forwarder answers with the real page title.

through the forwarder: All products | Books to Scrape - Sandbox

The forwarder handles both shapes a browser produces, a CONNECT tunnel for HTTPS and a plain request line for HTTP. It also gives you one place to swap the upstream proxy, which the next section uses.

Rotating Proxies in Selenium

Chrome reads --proxy-server once at startup, so a running browser cannot be moved to another proxy. Rotation in Selenium therefore means one of two things.

A new browser per proxy is the simple version. Each iteration picks an address from the pool, starts Chrome with it, does its work and quits:

import random
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

POOL = [
    "http://194.126.37.94:8080",
    "http://185.199.229.156:7492",
]

for attempt in range(4):
    proxy = random.choice(POOL)
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument(f"--proxy-server={proxy}")
    driver = webdriver.Chrome(options=options)
    try:
        driver.set_page_load_timeout(30)
        driver.get("https://books.toscrape.com/")
        print(f"run {attempt + 1} through {proxy}: {driver.title[:40]}")
    except Exception as error:
        print(f"run {attempt + 1} through {proxy}: {type(error).__name__}")
    finally:
        driver.quit()

The run against two local proxies prints one line per attempt:

run 1 through http://127.0.0.1:8892: All products | Books to Scrape - Sandbox
run 2 through http://127.0.0.1:8892: All products | Books to Scrape - Sandbox
run 3 through http://127.0.0.1:8891: All products | Books to Scrape - Sandbox
run 4 through http://127.0.0.1:8891: All products | Books to Scrape - Sandbox

Restarting Chrome costs a second or two, so this suits page-level work rather than a request-per-proxy pattern.

The other version keeps one browser and rotates behind it. Point Chrome at the forwarder from the previous section and let the forwarder pick the upstream, or buy a rotating endpoint where the provider changes the exit for you. A rotating proxy service hands out one hostname and changes the exit behind it, which is the same shape without the code.

Whichever you use, try and finally around the driver matter more here than anywhere else. A dead proxy raises inside driver.get, and without finally the loop leaves a Chrome process per failure.

How often to rotate

Rotating on every page is the reflex and it is usually wrong for a browser. A session that changes address between the login and the page behind it looks stranger to the site than one that keeps a single address for a hundred pages, and Selenium is generally used for exactly those multi-step flows. Sites that tie a session to an address will drop it outright.

Three patterns cover most work. One address per browser, held for the whole run, is the default for anything with a session. One address per batch, changing every few dozen pages, fits list-and-detail crawls where each page is independent. One address per page is for targets that count requests hard, and it costs a browser restart each time unless the rotation happens upstream. Providers sell the second and third as sticky sessions and per-request rotation, and the sticky window is usually somewhere between one and thirty minutes.

A scraper that survives a dead proxy

Pools contain dead addresses, and a page that fails on one proxy usually loads on the next. The version below walks a paginated catalogue, retries each page on a different proxy, and keeps the address it succeeded with next to the row, which is what you want in the CSV when a site starts refusing one exit.

import csv
import random

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

POOL = [
    "http://194.126.37.94:8080",
    "http://185.199.229.156:7492",
    "http://45.140.143.77:18080",
]


def driver_with(proxy):
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument(f"--proxy-server={proxy}")
    driver = webdriver.Chrome(options=options)
    driver.set_page_load_timeout(30)
    return driver


def fetch_page(url, attempts=3):
    """Open one page through a random proxy, moving to another proxy when one fails."""
    tried = []
    for _ in range(attempts):
        proxy = random.choice([p for p in POOL if p not in tried] or POOL)
        tried.append(proxy)
        driver = None
        try:
            driver = driver_with(proxy)
            driver.get(url)
            cards = driver.find_elements(By.CSS_SELECTOR, "article.product_pod")
            if not cards:                      # a challenge page has a body too
                raise RuntimeError("no cards on the page")
            return [{
                "title": c.find_element(By.CSS_SELECTOR, "h3 a").get_attribute("title"),
                "price": c.find_element(By.CSS_SELECTOR, ".price_color").text,
                "proxy": proxy,
            } for c in cards]
        except Exception as error:
            print(f"  {url.rsplit('/', 1)[-1]} through {proxy}: {type(error).__name__}, trying another proxy")
        finally:
            if driver:
                driver.quit()
    return []


books = []
for page in range(1, 4):
    books.extend(fetch_page(f"https://books.toscrape.com/catalogue/page-{page}.html"))

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["title", "price", "proxy"])
    writer.writeheader()
    writer.writerows(books)
print(f"saved {len(books)} books")

Run with one dead address in the pool, it reports the failure and carries on:

  page-1.html through http://127.0.0.1:8882: 20 books
  page-2.html through http://127.0.0.1:8889: WebDriverException, trying another proxy
  page-2.html through http://127.0.0.1:8881: 20 books
  page-3.html through http://127.0.0.1:8881: 20 books
saved 60 books

The empty-result check matters as much as the exception handler. A proxy that returns a challenge page raises nothing at all, and without that RuntimeError the scraper would write an empty page as a successful one.

Troubleshooting

Chrome reports proxy problems as page errors rather than exceptions, so a scraper that only catches Python errors writes empty rows instead of stopping.

Two Selenium habits help while you are in there. Wait for an element that only exists on the real page rather than trusting driver.get to have loaded content, which the guide on waiting for a page to load covers, and check what find_element returns before parsing, because a challenge page has a body too.

Conclusion

Setting a proxy in Selenium is one argument. Everything hard about it is authentication, and on Chrome 151 the popular answers no longer run. Credentials in the switch are dropped, the extension recipe does not load, selenium-wire fails on import, and Selenium’s own BiDi handler timed out against a proxy challenge. What worked was a proxy that does not ask for a password and a forty-line forwarder that answers on the browser’s behalf.

If the browser is only there for the proxy, it may not be needed at all. The Web Scraping API takes a URL, sends the request through its own datacenter or residential exits, renders the page when you ask for rendering, and returns the HTML, which removes both the driver and the credentials from your code.