Fetching a page gives you a string of HTML. Turning that string into fields you can store is a separate job, and Beautiful Soup is the shortest way to do it in Python. It is the parser most people learn first, and Python’s other scraping libraries trade that simplicity for speed or for a full crawling framework.
Getting Started with Beautiful Soup
The examples below run on Python 3.10 or newer, the floor the current requests and soupsieve releases declare. Use a virtual environment so this project’s library versions stay independent of everything else on the machine, and setting one up takes two commands.
Install Beautiful Soup
Beautiful Soup parses HTML that something else has already fetched. It builds a tree from the markup and gives you methods to search it, and it makes no HTTP requests of its own. The distribution on PyPI is named beautifulsoup4, the package you import is bs4, and PyPI’s bs4 is only a stub that pulls in the real one.
Install it together with the request library the examples use:
pip install beautifulsoup4 requestsWith both installed, everything below runs as pasted.
The Basic Structure of HTML Pages
Parsing starts with the page’s structure. HTML consists of tags, each with a specific role. Every page has a <head> containing the title, styles, and scripts, and a <body> containing the visible content.
Links live in <a> tags and running text in <p> tags, so knowing the structure tells you where to point a selector. Here is example.com’s markup with the styles cut out:
<html>
<head>
<title>Example Domain</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
<div>
<h1>Example Domain</h1>
<p>This domain is for use in illustrative examples in documents. You may use this
domain in literature without prior coordination or asking for permission.</p>
<p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>Tag and class names are chosen by whoever built the page, so selectors do not transfer between sites. Every new target starts with reading its markup, and the Elements tab in the browser’s DevTools (F12) shows that markup for any page.
Get an HTML from File or Website
Before anything can be parsed, the HTML has to come from somewhere, either a file on disk or a live request. Reading a saved copy, here an index.html next to the script, is the version to use while working out the selectors:
with open('index.html', 'r', encoding='utf-8') as file:
html_code = file.read()A live page needs a request library. requests, urllib and http.client all work, and requests is the one the rest of this guide uses:
import requests
response = requests.get('https://example.com', timeout=30)
response.raise_for_status()
html_code = response.contentThe timeout argument and the raise_for_status call both exist to stop the parser working on something that is not the page. Without a timeout, a target that accepts the connection and never answers hangs the script indefinitely, since Requests has no default timeout at all. Without raise_for_status, a 404 or a 503 is handed to the parser as if it were content, and the selectors then return nothing for a reason that looks like a layout change. The two statuses worth telling apart are 403, where the site refused this client and retrying the same request changes nothing, and 429, which means too many requests too fast and clears after waiting out the Retry-After header.
Passing response.content rather than response.text lets the parser read the character encoding the page declares. .text decodes using the charset from the Content-Type header, and when the header names none, Requests falls back to ISO-8859-1, so a page that declares UTF-8 only in a meta tag comes through with mangled non-ASCII characters.
Now the variable html_code holds the HTML of the page, ready for the parser.
Parse HTML Code With Beautiful Soup
A soup object represents the parsed document. The constructor takes the HTML and the name of the parser that will build the tree:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_code, 'html.parser')Every search in the rest of this guide runs against an object built this way.
Which Parser to Use
html.parser ships with Python and is the right default. lxml parses the same pages roughly a quarter to a third faster and needs a C extension installed. html5lib rebuilds the tree exactly the way a browser does and pays for that with about double html.parser’s time. Measured on two live pages, median of thirty parses each:
| Parser | Install | 49 KB catalogue page | 773 KB Wikipedia page | Broken markup |
|---|---|---|---|---|
html.parser | built in | 38 ms | 493 ms | closes unclosed tags by nesting them |
lxml | pip install lxml | 29 ms | 360 ms | closes unclosed tags as siblings |
html5lib | pip install html5lib | 71 ms | 1,018 ms | follows the browser’s recovery rules |
For one page nobody notices the 9 milliseconds. On a crawl of a hundred thousand pages, the gap between lxml and html5lib is most of a day of CPU time. The last column is the one that decides correctness rather than speed.
Find HTML Elements
Fourteen search methods cover the tree in every direction. In practice find, find_all, select and select_one do almost all of the work, and the rest exist for walking the tree when a selector cannot reach the element directly:
| Method | Description |
|---|---|
find(name, attrs, recursive, string, **kwargs) | First element matching the parameters, or None. |
find_all(name, attrs, recursive, string, limit, **kwargs) | All matching elements, as a list. |
select(selector, namespaces, limit, **kwargs) | All elements matching a CSS selector, as a list. |
select_one(selector, namespaces, **kwargs) | First element matching a CSS selector, or None. |
find_next(name, attrs, string, **kwargs) | Next match anywhere after this element in the document. |
find_all_next(name, attrs, string, limit, **kwargs) | All matches after this element. |
find_previous(name, attrs, string, **kwargs) | Previous match anywhere before this element. |
find_all_previous(name, attrs, string, limit, **kwargs) | All matches before this element. |
find_parent(name, attrs, **kwargs) | Nearest enclosing element that matches. |
find_parents(name, attrs, limit, **kwargs) | All enclosing elements that match. |
find_next_sibling(name, attrs, string, **kwargs) | Next match at the same nesting level. |
find_next_siblings(name, attrs, string, limit, **kwargs) | All following siblings that match. |
find_previous_sibling(name, attrs, string, **kwargs) | Previous matching sibling. |
find_previous_siblings(name, attrs, string, limit, **kwargs) | All preceding siblings that match. |
Only find and find_all take recursive. To restrict a CSS selector to direct children, put > in the selector itself. The full list of methods is in the official documentation.
By HTML Tag Name
The simplest search is by tag name. If every link on the page is wrapped in an <a> tag, find_all() returns all of them:
all_a_tags = soup.find_all('a')find_all returns a list. find() returns only the first match:
first_a_tags = soup.find('a')The search covers the whole document, including markup the browser never renders, so find_all('a') returns navigation, header and footer links along with the content ones. Scoping the search to a container first, soup.select('main a') instead of soup.find_all('a'), keeps the noise out.
By ID
An id is meant to appear once per page, which makes it the most stable attribute to select on:
element_by_id = soup.find(id='example_id')The parser does not enforce that uniqueness. Duplicate IDs parse without complaint, find returns the first and find_all returns them all, so two results from an ID search mean the page broke the rule, not the library.
By Class Name
Classes are the attribute most worth targeting, because developers assign them for styling and therefore assign them consistently across every card, row or listing on a page. The argument takes a trailing underscore, since class is a Python keyword:
elements_by_class = soup.find_all(class_='example_class')A tag with class="price col-md-4" matches find_all(class_='price'), because the filter matches any single class in the list rather than the whole attribute. Passing several classes at once is a trap, though. class_='featured card' matches only that exact string in that order, while the CSS selector .card.featured matches both classes in any order.
Using CSS Selectors
CSS selectors reach elements that need several conditions at once. Finding every <a> inside an element with the container class takes one expression:
selected_elements = soup.select('.container a')select accepts the same CSS selectors you test in the DevTools console, so a selector that works there works here without translation. select_one returns the first match alone.
Extract Data
A matched element yields its data in one of two ways, as the text between its tags or as the value of one of its attributes.
Extract Text Content
Running text sits between the opening and closing tags, and .text reads it:
paragraph_text = soup.find('p').textfind_all returns a list, not an element, so extracting text from every paragraph means iterating:
all_paragraphs = soup.find_all('p')
all_paragraph_texts = [paragraph.text for paragraph in all_paragraphs].text concatenates every text node inside the element, including the text of nested tags.
Extract Attribute Values
Links are the usual case. The address is not the tag’s text but its href attribute:
<a href="https://www.iana.org/domains/example">More information...</a>Square brackets read an attribute off a found element:
link_href = soup.find('a', href=True)['href']The same works for every link on the page:
all_links = soup.find_all('a', href=True)
all_link_hrefs = [link['href'] for link in all_links]The href=True filter is what keeps the square-bracket access from raising KeyError, since anchors without an href attribute otherwise land in the results too. The same bracket access reads any attribute off any tag.
Navigate and Filter the Parse Tree
The methods below earn their keep when the element you want has no usable class or ID of its own.
Navigate between Elements
When an element itself is anonymous but its neighbour is identifiable, start from the neighbour:
current_element = soup.find(id='example_id')Then take the next <p> at the same nesting level:
next_element = current_element.find_next_sibling('p')You can also find the previous sibling:
prev_element = current_element.find_previous_sibling('p')find_parent and find_parents climb the other way, toward the enclosing elements.
Handling Nested Elements
Dot notation reaches child elements in fewer characters than find. Any tag name reads as an attribute of its parent element.
Take a parent <div>:
outer_div = soup.find('div')Then access any child element directly:
inner_paragraph = outer_div.pouter_div.p is exactly outer_div.find('p'), the first match or None, so the same guard rules apply to it.
Filter Elements
Search methods also filter by text content. This finds every <p> whose text is exactly the given string:
filtered_elements = soup.find_all('p', string='desired_text')The argument is string. Older tutorials pass text instead, which still works on beautifulsoup4 4.15 and raises a DeprecationWarning saying so. The filter compares against the element’s .string, which is None for any tag with children, so <p>Hello <b>bold</b></p> never matches. When the text spans nested tags, loop over candidates and compare .get_text() instead.
Extract All Text
get_text() on the whole soup object dumps every piece of text on the page into one string:
all_text = soup.get_text()The dump includes the contents of <script> and <style> tags, since those are text nodes too. Pass strip=True and remove those tags first when the output should read like the page.
Make HTML Formatting Pretty
prettify() is a debugging tool. Print it when a selector returns nothing and you need to see the tree the parser actually built rather than the one DevTools shows:
pretty_html = soup.prettify()The output is re-indented one tag per line. The whitespace it adds becomes real text nodes if re-parsed, so parse the original, never the prettified copy.
Broken HTML, Missing Elements and JavaScript Pages
Real pages break in ways the clean examples above do not.
Missing or Broken HTML
Malformed markup gets repaired by the parser, not by Beautiful Soup itself, and each parser repairs it differently. An unclosed paragraph parses fine:
incomplete_html = '<p>This is a paragraph'
soup = BeautifulSoup(incomplete_html, 'html.parser')The differences start with chained mistakes. Fed <p>first<p>second<p>third, lxml and html5lib close each paragraph as a sibling, the same structure a browser builds, while html.parser nests each one inside the previous. On that markup find('p').find_next_sibling('p') returns the second paragraph under lxml and None under html.parser. A selector that works in the DevTools console but returns nothing in the script often traces back to this, because DevTools shows the browser’s tree rather than the parser’s, and html5lib is the parser that reproduces it.
Why find and select_one Return None
A search that finds nothing returns None, and the exception arrives one line later when something reads an attribute off it. AttributeError: 'NoneType' object has no attribute 'get_text' is the error Beautiful Soup code hits most often, and it means the selector matched nothing rather than that the library failed.
title = soup.select_one('h1.product-title')
print(title.get_text()) # AttributeError when the selector matches nothingThe guard is one if, and the message names the real cause:
title = soup.select_one('h1.product-title')
if title:
print(title.get_text(strip=True))
else:
print('title not found, the layout has probably changed')The distinction matters when a scraper runs unattended. A missing field on one page is normal, and a missing field on every page means the site changed and the selectors need updating. Counting how often a selector comes back empty is what tells the two apart.
Dynamic Content
A page whose content is built by JavaScript arrives as an empty shell, and no parser can read what was never in the markup. A browser has to render it first, which is what Selenium and Playwright do. With Selenium the handover looks like this:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from bs4 import BeautifulSoup
driver = webdriver.Chrome()
try:
driver.get('https://example.com')
WebDriverWait(driver, 10).until(
lambda d: d.execute_script('return document.readyState') == 'complete')
soup = BeautifulSoup(driver.page_source, 'html.parser')
finally:
driver.quit()The try/finally matters because every run without quit() leaves a Chrome and a chromedriver process behind, and the wait keeps page_source from being read before the page finishes loading. Both drivers have their own selector calls, so adding bs4 here pays off only when parsing code already exists and moving it would cost more than keeping it. A script written from scratch can use the driver’s selectors and skip the extra dependency.
A Complete Scraper
This script fetches a book catalogue page, parses every product card, and writes the results to CSV, with a guard at each point where a real site breaks. The HEADERS line sends a browser user agent, since some sites answer a bare python-requests client with an error page.
import csv
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
URL = "https://books.toscrape.com/catalogue/page-1.html"
HEADERS = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}
try:
response = requests.get(URL, headers=HEADERS, timeout=30)
response.raise_for_status()
except requests.RequestException as error:
raise SystemExit(f"request failed: {error}")
soup = BeautifulSoup(response.content, "html.parser")
cards = soup.select("article.product_pod")
if not cards:
raise SystemExit("no product cards found, the page layout has probably changed")
books = []
for card in cards:
title_tag = card.select_one("h3 a")
price_tag = card.select_one("p.price_color")
stock_tag = card.select_one("p.instock")
if not title_tag or not price_tag or not stock_tag:
continue
books.append({
"title": title_tag.get("title", "").strip(),
"price": price_tag.get_text(strip=True).removeprefix("\u00a3"),
"in_stock": "In stock" in stock_tag.get_text(),
"url": urljoin(URL, title_tag.get("href", "")),
})
with open("books.csv", "w", newline="", encoding="utf-8") as file:
writer = csv.DictWriter(file, fieldnames=["title", "price", "in_stock", "url"])
writer.writeheader()
writer.writerows(books)
print(f"{len(books)} of {len(cards)} cards parsed, written to books.csv")Run against the catalogue page it prints 20 of 20 cards parsed, and the first data row of the CSV reads A Light in the Attic,51.77,True,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html. The two counts are printed together on purpose. When they diverge, some cards are missing one of the three fields, which is the earliest signal that a layout has shifted.
Conclusion
Beautiful Soup parses HTML and does nothing else, which is the whole reason it is easy to learn. There is no request layer and no JavaScript engine, so the library has a small surface, and find, find_all, select and select_one cover most of what anyone needs. The other ten rows of the method table are for walking sideways and upward when a selector cannot reach an element directly.
That boundary is also where its limits are. Requests or another HTTP client has to fetch the page, and a page whose content is built in the browser arrives empty no matter which parser reads it, which is what Selenium and Playwright are for. For crawling many pages with retries and concurrency, Scrapy handles the parts Beautiful Soup deliberately leaves out.
For a page that arrives as ready HTML, requests plus bs4 runs about forty lines from URL to CSV, and the script above is all of them.
