Web Scraping with Python: from Fundamentals to Practice

A Python web scraper downloads a page, pulls the fields out of the HTML, handles the pages that only render inside a browser, and stores the result somewhere useful. requests and BeautifulSoup cover the first two steps for most sites, Playwright covers the third, and the standard library covers the fourth. Python is the most common language for the job, with Node.js as the main alternative, and the reason is the set of libraries below rather than the language itself.

Every section of this guide ends in a complete script that runs as written, against pages built for scraping practice, and the table right below maps each task to the library that handles it.

Choosing the Tools

The choice depends on what the page does rather than on what you already know.

The first row is where almost every project starts, and the rest of the table is what you add when the first row stops being enough.

Advantages of Using Python for Web Scraping

Python’s syntax reads close to plain English, so a scraper that downloads a page and extracts twenty fields fits on one screen, and the libraries for web scraping cover every step of the job without writing HTTP or HTML parsing code yourself. The community around them is large enough that most error messages you hit have already been answered.

Setting Up the Environment

The guide needs Python 3.10 or newer (3.14 is the current release), a code editor, a virtual environment, and a handful of packages. If you have all of that, skip to the next section.

Installing Python

Any Python from 3.10 up runs the code in this guide, and the current release is 3.14. Ubuntu and most other Linux distributions ship python3, and sudo apt install python3-pip python3-venv adds the package manager and the virtual environment module. On Windows, install the Python Install Manager from python.org or the Microsoft Store. It puts python and py on the command line, and the first python command downloads the current runtime by itself (py install default does the same explicitly). The prompt about adding a directory to PATH that appears at that point is optional and only matters for the versioned aliases such as python3.14. The older full installer with its “Add python.exe to PATH” checkbox is deprecated since 3.14. macOS users get Python from the same download page or from Homebrew. Open a new terminal and run python --version. A version number means everything after this point works the same on all three systems.

Choosing a Code Editor or IDE

A beginner is better off in a full IDE, which brings a debugger, a package manager view, and virtual environment handling in one window. PyCharm is built for Python and does all of that at the cost of memory. VS Code is lighter, needs the Python extension, and has Git built in. Sublime Text is a plain editor with syntax highlighting and a plugin for everything else, which suits people who already know their tools. Jupyter notebooks are handy for poking at a page interactively before the code becomes a script. The choice changes nothing about the code below.

Virtual Environment

A virtual environment keeps the packages of one project separate from every other project on the machine, so two scrapers can pin different versions of the same library. The venv module comes with Python on Windows and macOS, and on Ubuntu it is the python3-venv package installed above. Libraries without prebuilt wheels also need python3-dev to compile.

Create the environment in the project folder and activate it. Activation changes the shell so that python and pip point inside the environment.

python -m venv scraper-env

# Windows
scraper-env\Scripts\activate

# Linux and macOS
source scraper-env/bin/activate

A fresh environment holds only pip (and setuptools on older Python versions). Everything else in it is something you installed.

deactivate leaves the environment. pip freeze > requirements.txt writes the installed packages with their versions to a file, and pip install -r requirements.txt recreates the environment on another machine.

Installing Libraries

One command installs everything the scripts in this guide use, and Playwright needs a second one to download its browser.

pip install requests "httpx[http2]" beautifulsoup4 lxml pandas openpyxl playwright selenium aiohttp newspaper4k readability-lxml
playwright install chromium

requests and httpx download pages, beautifulsoup4 and lxml parse them, pandas and openpyxl write CSV and Excel files, playwright and selenium drive a browser, aiohttp sends requests concurrently, and the last two extract article text.

The list will look different on your machine as versions move, and that is fine as long as pip list shows the packages above.

Getting HTML Code of a Page Using HTTP Requests

The examples use Scrape This Site, a set of pages built for practicing this. The first one, Countries of the World, lists 250 countries as cards, and each card holds the name in an h3.country-name, and the capital, population, and area in three span elements with matching class names. It is plain HTML, so an HTTP request is all it takes.

Fetching a Page with Requests

requests sends the request, follows redirects, and decodes the body into a string. Send a browser User-Agent from the first request, because many sites answer the default python-requests/2.34.2 with a 403, and call raise_for_status() so that an error page raises an exception instead of being parsed as if it were data.

import requests

URL = "https://www.scrapethissite.com/pages/simple/"
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}

response = requests.get(URL, headers=HEADERS, timeout=30)
response.raise_for_status()  # raises on 4xx and 5xx instead of letting you parse an error page

html = response.text
print(response.status_code, len(html), "characters")
print(html[:120])

The page is about 200 KB of HTML.

200 203338 characters
<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>Countries of the World: A Simple Example

The timeout argument matters too. Without it a request to a stalled server waits forever, and a scraper that loops over a thousand URLs stops at the first one that hangs. response.text is the body decoded with the charset the server declared, and response.content is the raw bytes, which is what you want for images, PDFs, or a page whose declared charset is wrong.

Sessions, Headers, and Authentication

A Session reuses the TCP connection and keeps cookies between requests, which is what a site expects from one visitor. Headers set on the session go out with every request, and the User-Agent is the one to set first. Pages behind HTTP Basic authentication take the credentials through the auth argument, and the same session pattern carries login cookies for form-based logins.

import requests
from requests.auth import HTTPBasicAuth

HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}

with requests.Session() as session:  # one TCP connection and one cookie jar for every request below
    session.headers.update(HEADERS)
    response = session.get("https://httpbin.org/basic-auth/demo/secret", auth=HTTPBasicAuth("demo", "secret"), timeout=30)
    print(response.status_code, response.json())

The test endpoint confirms the credentials it received.

200 {'authenticated': True, 'user': 'demo'}

For a login form, post the credentials with session.post() once, and every later session.get() carries the session cookie the site set in response.

The Same Request with httpx

httpx has the same interface as requests and adds HTTP/2, which most sites serve to browsers, and an async client that appears later in this guide. Install it as httpx[http2] for the HTTP/2 support.

import httpx

URL = "https://www.scrapethissite.com/pages/simple/"
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}

with httpx.Client(headers=HEADERS, http2=True, timeout=30) as client:
    response = client.get(URL)
    print(response.status_code, response.http_version, len(response.text), "characters")

The practice site negotiates HTTP/2 and returns the same page.

200 HTTP/2 203338 characters

Either library works for the rest of this guide. requests has more tutorials written about it, httpx saves a second dependency when the project also needs async requests.

Parsing HTML Code of a Page

The HTML is a string, and the fields are somewhere inside it. Open the page in a browser, press F12, and use the element picker to click on a country name. DevTools highlights the h3.country-name element and, around it, the div.country card with the three span elements. Those class names are the selectors the scripts below use. Regular expressions, CSS selectors, and XPath each get the same data out, and two libraries built for article pages get text out of pages that have no stable structure at all.

Extracting Data with Regular Expressions

Regular expressions treat the page as text, which makes them a poor fit for nested HTML and a good fit for flat patterns such as email addresses, phone numbers, or one tag with a fixed class. The standard re module needs no installation.

import re
import requests

URL = "https://www.scrapethissite.com/pages/simple/"
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}

html = requests.get(URL, headers=HEADERS, timeout=30).text

capitals = re.findall(r'<span class="country-capital">([^<]+)</span>', html)
print(len(capitals), capitals[:5])

emails = re.findall(r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b", html)
print(emails)

All 250 capitals come out, and the page has no email addresses.

250 ['Andorra la Vella', 'Abu Dhabi', 'Kabul', 'St. John's', 'The Valley']
[]

The fourth capital shows the cost of working on raw text. The apostrophe in St. John’s arrives as the numeric HTML entity for the character, and a regular expression hands it over undecoded. An HTML parser returns St. John's.

Parsing HTML Elements using CSS Selectors with Beautiful Soup

BeautifulSoup parses the document into a tree and lets you address elements with the same CSS selectors DevTools shows. select() returns every match, select_one() the first, and get_text(strip=True) the text without surrounding whitespace. The loop below builds one dictionary per country, which is the shape every storage option in this guide accepts.

import requests
from bs4 import BeautifulSoup

URL = "https://www.scrapethissite.com/pages/simple/"
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}

response = requests.get(URL, headers=HEADERS, timeout=30)
soup = BeautifulSoup(response.text, "html.parser")

countries = []
for card in soup.select("div.country"):
    countries.append({
        "name": card.select_one("h3.country-name").get_text(strip=True),
        "capital": card.select_one("span.country-capital").get_text(strip=True),
        "population": int(card.select_one("span.country-population").get_text(strip=True)),
        "area_km2": float(card.select_one("span.country-area").get_text(strip=True)),
    })

print(len(countries), "countries")
print(countries[:2])

Numbers are converted where they are extracted, so the rest of the pipeline never sees strings pretending to be numbers.

250 countries
[{'name': 'Andorra', 'capital': 'Andorra la Vella', 'population': 84000, 'area_km2': 468.0}, {'name': 'United Arab Emirates', 'capital': 'Abu Dhabi', 'population': 4975593, 'area_km2': 82880.0}]

html.parser is the parser from the standard library. Passing "lxml" instead makes BeautifulSoup faster on large documents once lxml is installed (184 ms against 257 ms for this 200 KB page in my run), with the same code otherwise. The BeautifulSoup tutorial covers navigation between elements, attributes, and malformed HTML.

Extracting Data with XPath and lxml

lxml parses the same HTML into a tree that you query with XPath, a path language that can express conditions CSS cannot, such as “the span that follows a strong element with this text”. Its normalize-space() function trims the whitespace the practice site puts around every value. The card selector below matches the full class attribute, because contains(@class, "country") would also match the country-info blocks inside each card.

import requests
from lxml import html

URL = "https://www.scrapethissite.com/pages/simple/"
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}

tree = html.fromstring(requests.get(URL, headers=HEADERS, timeout=30).text)

countries = []
for card in tree.xpath('//div[@class="col-md-4 country"]'):
    countries.append({
        "name": card.xpath('normalize-space(.//h3[@class="country-name"])'),
        "capital": card.xpath('normalize-space(.//span[@class="country-capital"])'),
        "population": int(card.xpath('normalize-space(.//span[@class="country-population"])')),
        "area_km2": float(card.xpath('normalize-space(.//span[@class="country-area"])')),
    })

print(len(countries), "countries")
print(countries[:2])

The output is identical to the BeautifulSoup version, so the choice is about the query language. Which one to use is covered in XPath vs CSS selectors, and matching elements by their text, the one thing CSS cannot do, in selecting elements by text in XPath.

Extract Content from News Articles with Newspaper4k

News pages are a poor fit for selectors, because every site is different and the same site changes its layout. The newspaper library guesses the title, authors, date, and body text from the structure of the page. The original package, newspaper3k, stopped at version 0.2.8 years ago, and on a current installation it fails at import with ImportError: lxml.html.clean module is now a separate project lxml_html_clean. The maintained fork is newspaper4k (0.9.6 at the time of writing), and it keeps the same import path, so old code needs only the new package name.

from newspaper import Article  # pip install newspaper4k

article = Article("https://www.nasa.gov/news-release/nasas-artemis-ii-crew-set-to-receive-congressional-space-medal-of-honor/")
article.download()
article.parse()

print(article.title)
print(article.authors, article.publish_date)
print(article.text[:200])

A NASA press release comes back as title, author, timestamp, and body text, with no selectors written.

NASA’s Artemis II Crew Set to Receive Congressional Space Medal of Honor
['Gerelle Q. Dodson'] 2026-08-21 17:06:36+00:00
President Donald J. Trump will award each of NASA’s Artemis II crew members the Congressional Space Medal of Honor at 11 a.m. EDT on Friday, Aug. 28, during a ceremony at the agency’s Johnson Space Ce

The library is built for articles. On a product page or a listing it returns whatever block of text looks most like prose, which is rarely what you want.

Extract the Main Content with Readability

readability-lxml is the Python port of the algorithm behind browser reader modes. It takes HTML you already downloaded and returns the main content block as clean HTML, with navigation, sidebars, and footers removed. The package name matters, because pip install readability installs an unrelated text-statistics library.

import requests
from readability import Document  # pip install readability-lxml

URL = "https://www.nasa.gov/news-release/nasas-artemis-ii-crew-set-to-receive-congressional-space-medal-of-honor/"
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}

html = requests.get(URL, headers=HEADERS, timeout=30).text
doc = Document(html)

print(doc.title())
main_html = doc.summary()  # the article body as clean HTML, navigation and boilerplate removed
print(len(html), "characters in the page,", len(main_html), "in the extracted body")

The same press release shrinks from a quarter of a megabyte of page to two thousand characters of article.

NASA’s Artemis II Crew Set to Receive Congressional Space Medal of Honor - NASA
263782 characters in the page, 2000 in the extracted body

newspaper4k gives you fields, readability-lxml gives you the body as HTML you can parse further with BeautifulSoup. Both stop being useful the moment the data is a table or a price rather than an article.

Handling Dynamic Web Pages

Everything so far worked because the data was in the HTML the server sent. Many pages send an empty shell instead and fill it with JavaScript after load, usually by calling a JSON endpoint (the pattern is called AJAX). The practice site has a page like that, a table of Oscar-winning films whose rows appear only after you click a year. requests receives the shell, one header row and nothing else. The quickest test for any page is to compare View Source, which shows what the server sent, with Inspect, which shows the page after JavaScript ran. Data that appears only in the second is dynamic.

One way through is the JSON endpoint the page calls. Find it in DevTools, on the Network tab under Fetch/XHR, and request it directly, which is faster than any browser and is covered in scraping dynamic content in Python. The other way is a real browser without a window that executes the JavaScript and hands you the finished page, and that is what the two scripts below do.

Playwright

Playwright drives Chromium, Firefox, and WebKit through one API, waits for elements to appear instead of sleeping, and installs its own browser builds with playwright install chromium. The script clicks the 2015 tab, waits for the rows that the AJAX request inserts, and reads them.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://www.scrapethissite.com/pages/ajax-javascript/", wait_until="domcontentloaded")
    page.click("a.year-link[id='2015']")            # the table is empty until this click fires an AJAX request
    page.wait_for_selector("tr.film")                # wait for the rows the request inserts

    films = []
    for row in page.query_selector_all("tr.film"):
        films.append({
            "title": row.query_selector("td.film-title").inner_text().strip(),
            "nominations": int(row.query_selector("td.film-nominations").inner_text()),
            "awards": int(row.query_selector("td.film-awards").inner_text()),
        })
    browser.close()

print(len(films), "films")
print(films[:2])

Sixteen films for 2015, none of which exist in the HTML requests receives.

16 films
[{'title': 'Spotlight', 'nominations': 6, 'awards': 2}, {'title': 'Mad Max: Fury Road', 'nominations': 10, 'awards': 6}]

wait_for_selector is the line that makes this reliable. A fixed sleep either wastes seconds or fails on a slow network, and waiting for the element you are about to read does neither. Pagination, logins, screenshots, and the async API are in the Playwright scraping guide and the walkthrough of Playwright with Python.

Selenium

Selenium is the older browser automation tool, and it drives the Chrome that is already installed on the machine. Since version 4.6 it downloads a matching driver by itself, so the setup is one pip install. --headless=new runs the full Chrome without a window (the old --headless flag started a separate, reduced implementation of the browser), and WebDriverWait does what wait_for_selector does in Playwright.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

options = Options()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)  # Selenium Manager downloads a matching chromedriver on first run

driver.get("https://www.scrapethissite.com/pages/ajax-javascript/")
driver.find_element(By.CSS_SELECTOR, "a.year-link[id='2015']").click()
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "tr.film")))

films = []
for row in driver.find_elements(By.CSS_SELECTOR, "tr.film"):
    films.append({
        "title": row.find_element(By.CSS_SELECTOR, "td.film-title").text.strip(),
        "nominations": int(row.find_element(By.CSS_SELECTOR, "td.film-nominations").text),
        "awards": int(row.find_element(By.CSS_SELECTOR, "td.film-awards").text),
    })
driver.quit()

print(len(films), "films")
print(films[:2])

Same sixteen films.

16 films
[{'title': 'Spotlight', 'nominations': 6, 'awards': 2}, {'title': 'Mad Max: Fury Road', 'nominations': 10, 'awards': 6}]

Selenium makes sense when a project already uses it or when the team knows it from testing, and the Selenium scraping guide and XPath in Selenium cover it in depth. For new code, Playwright is the better default. Its waits are built in, it bundles the browsers it was tested with, and its API is the same in sync and async form. Pyppeteer, the Python port of Puppeteer, still installs (its last release is 2.0.0), but its repository has been idle for more than two years at the time of writing, so treat it as a library for maintaining existing code rather than starting new scrapers.

Data Storage

The scripts above end with a list of dictionaries, and that shape goes into a file or a database with a few lines. CSV and Excel are for people who open the data in a spreadsheet, JSON is for other programs, and SQLite is for anything you will query or update later.

Storing Data in CSV and Excel

pandas writes the list to CSV in one call and to Excel in another, with the openpyxl package doing the Excel part. The csv module from the standard library does the same for CSV without the pandas dependency and writes row by row, which matters when the scraper produces millions of rows and you do not want them all in memory first.

import csv
import requests
import pandas as pd
from bs4 import BeautifulSoup

URL = "https://www.scrapethissite.com/pages/simple/"
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}


def scrape_countries() -> list[dict]:
    soup = BeautifulSoup(requests.get(URL, headers=HEADERS, timeout=30).text, "html.parser")
    return [{
        "name": card.select_one("h3.country-name").get_text(strip=True),
        "capital": card.select_one("span.country-capital").get_text(strip=True),
        "population": int(card.select_one("span.country-population").get_text(strip=True)),
        "area_km2": float(card.select_one("span.country-area").get_text(strip=True)),
    } for card in soup.select("div.country")]


countries = scrape_countries()

# pandas: one call per format; to_excel needs the openpyxl package
df = pd.DataFrame(countries)
df.to_csv("countries.csv", index=False, encoding="utf-8")
df.to_excel("countries.xlsx", index=False)

# csv module from the standard library: no extra install, row by row
with open("countries_csv_module.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=countries[0].keys())
    writer.writeheader()
    writer.writerows(countries)

print(df.head(3))

Three files appear next to the script, and the DataFrame preview shows the typed columns.

name           capital  population  area_km2
0               Andorra  Andorra la Vella       84000     468.0
1  United Arab Emirates         Abu Dhabi     4975593   82880.0
2           Afghanistan             Kabul    29121286  647500.0

newline="" in the open() call is not optional on Windows. Without it the csv module writes a blank line after every row.

Storing Data in JSON Format

JSON keeps the nesting that CSV flattens, so a country with a list of cities stays a country with a list of cities. The standard json module writes the whole list as one document, and pandas writes JSON Lines, one object per line, which log tools and data pipelines read without loading the file at once. Reading JSON back in Python is the mirror image of this.

import json
import requests
import pandas as pd
from bs4 import BeautifulSoup

URL = "https://www.scrapethissite.com/pages/simple/"
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}

soup = BeautifulSoup(requests.get(URL, headers=HEADERS, timeout=30).text, "html.parser")
countries = [{
    "name": card.select_one("h3.country-name").get_text(strip=True),
    "capital": card.select_one("span.country-capital").get_text(strip=True),
    "population": int(card.select_one("span.country-population").get_text(strip=True)),
} for card in soup.select("div.country")]

with open("countries.json", "w", encoding="utf-8") as file:
    json.dump(countries, file, ensure_ascii=False, indent=2)

pd.DataFrame(countries).to_json("countries.jsonl", orient="records", lines=True, force_ascii=False)

print(open("countries.json", encoding="utf-8").read()[:150])

ensure_ascii=False keeps names like “São Tomé” readable instead of escaping them.

[
  {
    "name": "Andorra",
    "capital": "Andorra la Vella",
    "population": 84000
  },
  {
    "name": "United Arab Emirates",
    "capital": "A

indent=2 makes the file readable at the cost of size, and the JSON Lines file skips it so that each line stays one record.

Storing Data in SQL Databases

SQLite ships with Python, needs no server, and stores the whole database in one file, which makes it the right first database for a scraper. The script creates the table if it is missing, inserts all rows in one executemany call, and runs a query to prove the data is there. executemany wants a sequence per row, so the scraping loop below builds tuples in column order rather than dictionaries.

import sqlite3
import requests
from bs4 import BeautifulSoup

URL = "https://www.scrapethissite.com/pages/simple/"
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}

soup = BeautifulSoup(requests.get(URL, headers=HEADERS, timeout=30).text, "html.parser")
countries = [(
    card.select_one("h3.country-name").get_text(strip=True),
    card.select_one("span.country-capital").get_text(strip=True),
    int(card.select_one("span.country-population").get_text(strip=True)),
    float(card.select_one("span.country-area").get_text(strip=True)),
) for card in soup.select("div.country")]

connection = sqlite3.connect("countries.db")  # the file is created on first connect
cursor = connection.cursor()
cursor.execute("""CREATE TABLE IF NOT EXISTS countries (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT,
    capital TEXT,
    population INTEGER,
    area_km2 REAL)""")
cursor.executemany("INSERT INTO countries (name, capital, population, area_km2) VALUES (?, ?, ?, ?)", countries)
connection.commit()

cursor.execute("SELECT name, population FROM countries ORDER BY population DESC LIMIT 3")
print(cursor.fetchall())
connection.close()

The query answers a question the CSV file could not without opening a spreadsheet.

[('China', 1330044000), ('India', 1173108018), ('United States', 310232863)]

Run the script twice and the table holds 500 rows, because nothing stops duplicates. A UNIQUE constraint on name plus INSERT OR REPLACE turns the second run into an update, and that small change is what makes a scraper safe to schedule. PostgreSQL and MySQL take the same pattern with their own driver packages (psycopg and mysql-connector-python), %s placeholders instead of ?, and their own auto-increment syntax in the CREATE TABLE statement.

Multithreading, Async Requests, and Proxies

One request at a time is fine for one page and slow for a thousand, because the script spends almost all of its time waiting for the network. Threads and async code both fix that by waiting for many responses at once. The examples fetch four pages of a paginated table of hockey teams, 25 rows each, and count the rows, which is enough to see the pattern.

Threads with concurrent.futures

ThreadPoolExecutor runs an ordinary function in several threads and collects the results in order. Nothing in the function changes, the executor calls it once per URL. Keep max_workers small, four to eight, since a site sees each thread as another concurrent visitor.

from concurrent.futures import ThreadPoolExecutor

import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}
PAGES = [f"https://www.scrapethissite.com/pages/forms/?page_num={n}&per_page=25" for n in range(1, 5)]


def scrape_page(url: str) -> int:
    soup = BeautifulSoup(requests.get(url, headers=HEADERS, timeout=30).text, "html.parser")
    return len(soup.select("tr.team"))  # one row per hockey team


with ThreadPoolExecutor(max_workers=4) as executor:  # four pages download at the same time
    rows_per_page = list(executor.map(scrape_page, PAGES))

print(rows_per_page, "rows,", sum(rows_per_page), "in total")

Four pages, downloaded together, parsed one after another.

[25, 25, 25, 25] rows, 100 in total

The parsing still runs in Python one page at a time, which is fine here, because downloading takes far longer than parsing.

Async Requests with asyncio and aiohttp

Asynchronous requests with asyncio do the same with one thread. Every await hands control back to the event loop while the response is in flight, and asyncio.gather starts all requests together. aiohttp is the async HTTP client most often paired with it, and its ClientSession plays the role of requests.Session.

import asyncio

import aiohttp
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}
PAGES = [f"https://www.scrapethissite.com/pages/forms/?page_num={n}&per_page=25" for n in range(1, 5)]


async def scrape_page(session: aiohttp.ClientSession, url: str) -> int:
    async with session.get(url) as response:
        html = await response.text()
    return len(BeautifulSoup(html, "html.parser").select("tr.team"))


async def main() -> None:
    async with aiohttp.ClientSession(headers=HEADERS) as session:
        rows_per_page = await asyncio.gather(*(scrape_page(session, url) for url in PAGES))
    print(rows_per_page, "rows,", sum(rows_per_page), "in total")


asyncio.run(main())

Same result, and the same shape scales to hundreds of URLs with an asyncio.Semaphore to cap how many are in flight.

[25, 25, 25, 25] rows, 100 in total

httpx offers the same thing with the interface you already saw, which is the reason to pick it when one project mixes sync and async code.

import asyncio

import httpx
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}
PAGES = [f"https://www.scrapethissite.com/pages/forms/?page_num={n}&per_page=25" for n in range(1, 5)]


async def main() -> None:
    async with httpx.AsyncClient(headers=HEADERS, timeout=30) as client:
        responses = await asyncio.gather(*(client.get(url) for url in PAGES))
    rows_per_page = [len(BeautifulSoup(r.text, "html.parser").select("tr.team")) for r in responses]
    print(rows_per_page, "rows,", sum(rows_per_page), "in total")


asyncio.run(main())

Threads suit code that already exists and uses requests. Async suits new code that will fetch thousands of pages, and it pairs with Playwright’s async API when those pages need a browser. Either way, concurrency multiplies your request rate, and the site notices. The four-page example is polite. Forty workers against one site is how you get blocked.

Using Proxies

A proxy routes requests through another IP address, which spreads the load across addresses and lets you appear to come from the country the site serves. requests takes a proxies dictionary, and the address belongs in an environment variable rather than in the code, because it usually carries a password.

import os
import requests

# "http://user:password@host:port"; leave the variable unset to connect directly
PROXY = os.environ.get("SCRAPER_PROXY")
proxies = {"http": PROXY, "https": PROXY} if PROXY else None

response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=30)
print(response.json()["origin"])

The endpoint echoes the address the request arrived from, so a working proxy shows its own IP instead of yours. Authentication, SOCKS proxies, and per-request rotation are in proxies with Python requests, and the choice between datacenter, residential, and mobile pools in proxies for web scraping and rotating proxies.

Errors, Blocks, and Layout Changes

Requests fail, sites refuse to answer, and pages change under their selectors, and that is also the order in which a new scraper meets them.

Error Handling

A scraper that runs over a list of URLs will hit a 404, a timeout, and a dropped connection, and one unhandled exception ends the whole run. requests raises a different exception class for each case, so a try block can log the failure and continue.

import requests

HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}
url = "https://www.scrapethissite.com/pages/this-page-does-not-exist/"

try:
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    print(len(response.text), "characters")
except requests.exceptions.HTTPError as error:
    print(f"HTTP error: {error}")
except requests.exceptions.ConnectionError as error:
    print(f"Connection error: {error}")
except requests.exceptions.Timeout as error:
    print(f"Timeout: {error}")
except requests.exceptions.RequestException as error:
    print(f"Request failed: {error}")

A missing page becomes one logged line instead of a crash.

HTTP error: 404 Client Error: Not Found for url: https://www.scrapethissite.com/pages/this-page-does-not-exist/

Timeouts and connection errors are usually worth a retry with a pause, a 404 is not, and a 429 means the site asked you to slow down. The guide to retrying failed requests has the backoff code for each case.

Web Scraping Challenges

Sites that do not want to be scraped check the TLS fingerprint of the HTTP client, the IP address and its request rate, the headers, and the JavaScript-visible properties of the browser, and they answer a scraper with a 403, a 429, or a challenge page. The full list of checks and the countermeasure for each is in web scraping without getting blocked. The short version is that proxies, browser headers, and a real browser profile handle most sites, and all of it has to be maintained.

The alternative is to send the URL to a scraping API and get the page back. Our Web Scraping API fetches the page through a datacenter or residential proxy, renders JavaScript when asked, and applies the same CSS selectors you would use in BeautifulSoup through extractRules, so the response arrives as fields rather than HTML.

import os
import requests

response = requests.post(
    "https://api.hasdata.com/scrape/web",
    headers={"x-api-key": os.environ["HASDATA_API_KEY"], "Content-Type": "application/json"},
    json={
        "url": "https://www.scrapethissite.com/pages/simple/",
        "proxyType": "datacenter",
        "jsRendering": True,
        "extractRules": {
            "name": "h3.country-name",
            "capital": "span.country-capital",
            "population": "span.country-population",
        },
    },
    timeout=120,
)
print(response.status_code)

data = response.json()["extractedData"]
countries = [
    {"name": name.strip(), "capital": capital.strip(), "population": int(population)}
    for name, capital, population in zip(data["name"], data["capital"], data["population"])
]
print(len(countries), "countries")
print(countries[:2])

The same 250 countries, with the request, the proxy, and the rendering on the API’s side.

200
250 countries
[{'name': 'Andorra', 'capital': 'Andorra la Vella', 'population': 84000}, {'name': 'United Arab Emirates', 'capital': 'Abu Dhabi', 'population': 4975593}]

This call costs ten credits (JavaScript rendering through a datacenter proxy). Without rendering it is one credit, and only successful responses are billed. The response also carries the full HTML in content, so the BeautifulSoup code above works on it unchanged.

Changes in HTML Document Structure

Selectors break when a site redesigns, renames a class, or moves a block, and nothing in the scraper notices until the output is empty. Validate the output on every run, so that zero rows or a column of None fails loudly instead of landing in the database. Prefer selectors tied to meaning over selectors tied to layout, an id or a data- attribute over div:nth-child(3). And when a site offers a JSON endpoint or a feed, scrape that, because APIs change far less often than markup. For pages you cannot pin down at all, the API above also accepts aiExtractRules, a description of the fields you want instead of selectors, which is slower per request but survives redesigns.

Conclusion

The order of the sections is the order of a real project. Check whether the data is in the HTML requests receives, and if it is, BeautifulSoup or lxml and a csv or sqlite3 writer finish the job in one script. If the table is empty until JavaScript runs, look for the JSON endpoint first and reach for Playwright second. Add threads or asyncio when the URL list grows past a few dozen, keep the concurrency low enough that the site does not notice, and move the proxy and rendering work to an API when maintaining it costs more than the data is worth. Every script in this guide is a starting point in that sequence, and each one runs as it stands.