XPath for Web Scraping, from Syntax to Selenium

Finding the data comes before extracting it, and XPath is the query language for that job. It addresses any node in the DOM (Document Object Model) the browser builds from a page, walks the tree in both directions, and works in every scraping stack, from browser DevTools to Selenium, lxml, and Scrapy. This guide covers the syntax, both path types, the operators and axes with a working example for each, and the same queries running in Selenium. Every expression here was run against a live page or the sample page below, and the results are shown as they came back.

Syntax of XPath

XML Path, commonly known as XPath, is a query language for XML and HTML documents. An expression describes a route to the elements you want, and the general shape is:

//tag_name[@attribute = "value"]/child_tag

The // means “search at any depth”, the tag name names the node, @ starts an attribute, and the bracketed condition filters the matches. A real one from the sample shop below looks like this:

//div[@class="product-list"]/h3

It reads as any div with the class product-list, then its h3 children.

Types of XPath

There are 2 types of XPath in practice, absolute and relative. The examples use a small shop page with this HTML:

<!DOCTYPE html>
<html>
    <head>
        <title>A sample shop</title>
    </head>
    <body>
        <div class="product-item">
            <img src="item1.jpg">
            <div class="product-list">
                <h3>Pen</h3>
                <span class="price">10$</span>
                <a href="item1.html" class="button">Buy</a>
            </div>
        </div>
        <div class="product-item">
            <img src="item2.jpg">
            <div class="product-list">
                <h3>Book</h3>
                <span class="price">20$</span>
                <a href="item2.html" class="button">Buy</a>
            </div>
        </div>
    </body>
</html>

Two products, and each carries a name, a price, and a buy link. Everything below runs against this page.

Absolute XPath

An absolute XPath spells out the whole route from the root. For the first product name:

The full route to it reads:

/html/body/div[1]/div/h3

DevTools writes these for you. Inspect the element (F12, then Ctrl+Shift+C or the picker button):

Then right-click the highlighted line and choose Copy, Copy full XPath:

The result can be checked in the Console with $x():

One caveat before copying these into a scraper. An absolute path encodes every step, so the first layout change anywhere along the route breaks it. It answers “where exactly is this one element right now”, and little else.

Relative XPath

A relative XPath anchors on a recognizable feature instead of the full route, which survives layout changes around it. This one finds the same element as the absolute path above:

//*[@class="product-list"]/h3

Run against the sample page, it returns both product names, because both cards match the class:

[<h3>Pen</h3>, <h3>Book</h3>]

Relative paths start anywhere in the DOM, read shorter, and are what scraping code actually uses. The absolute form stays useful mainly as DevTools output to trim down.

XPath VS CSS Selectors

The practical difference is direction. A CSS selector reads down the tree, while XPath also walks up and sideways, to parents, ancestors, and preceding elements, and matches on text content, which CSS can’t do at all. Modern CSS closes part of the gap with :has(), and our CSS selectors cheat sheet covers that side. For picking between them case by case, there’s a separate XPath vs CSS selectors comparison. The short version is that CSS reads cleaner going down, and XPath takes over once the anchor is below or beside what you need.

Using XPath in Selenium

Selenium takes XPath through the By class, with two methods that differ in how they fail:

1. find_element returns the first match and raises NoSuchElementException when there is none.
2. find_elements returns a list of all matches, and an empty list when there are none, which makes it the safer default for scraping.

Both work against a live catalogue page:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

options = Options()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)

driver.get("https://books.toscrape.com/")

first = driver.find_element(By.XPATH, "//article[@class='product_pod']//h3/a")
print(first.get_attribute("title"))

titles = driver.find_elements(By.XPATH, "//article[@class='product_pod']//h3/a")
print(len(titles))

print(first.get_attribute("title"))

That prints A Light in the Attic and 20, one link per product card.

The axes work in Selenium unchanged. This one walks up from a known title to its card and back down to the price, and returns £51.77:

price = driver.find_element(
    By.XPATH,
    "//h3/a[@title='A Light in the Attic']/ancestor::article//p[@class='price_color']"
)
print(price.text)

driver.quit()

Waits, clicks and the rest of the loop around these expressions belong to the Selenium scraping guide rather than to this page.

Dynamic XPath in Selenium

Attributes alone stop working when pages get dynamic, and that’s what the operators and axes are for. The table sums up the toolbox:

Each row below gets its own section with the markup it selects.

XPath Using Logical Operators OR and AND

Logical operators combine conditions inside the brackets. XPath has 2 of them, or and and, and both are case-sensitive, so writing OR or AND fails.

Logical Operator OR

The query matches elements that satisfy the first condition, the second, or both:

On the sample page, this asks for elements that are a price or a buy button:

//span[@class="price" or @class="button"]

That returns both price spans, 10$ and 20$. The @class="button" half matched nothing on its own (the buttons are a tags, and the expression asks for span), and the query still succeeded through the first condition, which is the point of or.

Logical Operator AND

With and, an element must satisfy both conditions:

This finds a link that is a buy button and points at the first item:

//a[@class="button" and starts-with(@href, "item1")]

One element comes back, the Pen’s buy link. Swap and for or in the same query and both buttons return.

XPath using Starts-With()

starts-with() matches on how an attribute or the text opens, which helps when the rest of the value is generated. On the sample page:

//h3[starts-with(text(), 'P')]

That returns the Pen and skips the Book. The same function on an attribute handles the dynamic-suffix case, like classes that render as price-a8f3 one build and price-c07d the next:

//span[starts-with(@class, 'price')]

The stable prefix is the anchor, and the generated tail doesn’t matter anymore.

XPath using Index

An index picks one match by position, and the parentheses are the part people trip on:

(//*[@class="product-list"]/h3)[1]

That returns the Pen alone. Written without the parentheses, //*[@class="product-list"]/h3[1] means something else, “each h3 that is the first h3 inside its parent”, which on this page matches both products, and that’s rarely what you meant. Wrap the whole expression before indexing when you mean “the first result overall”.

XPath using Following

The following axis selects everything after the known node in document order, at any level:

//h3[text()='Pen']/following::span

Both price spans come back, 10$ and 20$, because both sit after the Pen heading in the document, even though the second one belongs to a different card.

XPath using Following-Sibling

following-sibling narrows that to nodes sharing the same parent:

//h3[text()='Pen']/following-sibling::span

Now only 10$ returns, the price inside the Pen’s own card. This pair, following against following-sibling, is usually the difference between “somewhere below on the page” and “in the same block”, and the sibling version is what price-next-to-name extraction wants.

XPath using Preceding

preceding looks backward through the whole document:

(//span[@class='price'])[1]/preceding::img

That returns the first card’s cover, and swapping the 1 for a 2 returns both, since both images appear earlier in the document than the second price. The brackets matter here for the reason the index section gives. Written as //span[@class='price'][1], the predicate applies inside each parent, so it matches both prices and the axis step then returns both images.

XPath using Preceding-Sibling

The backward twin of following-sibling, staying under the same parent:

//a[@class='button']/preceding-sibling::h3

Each buy button’s sibling heading comes back, Pen and Book, one per card. The preceding-sibling answer covers this axis on its own.

XPath using Child

child selects direct children only:

//div[@class="product-list"]/child::h3

Both product names return. /child::h3 and plain /h3 mean the same thing, so the explicit axis mostly appears in generated code and in queries where the axis is a variable.

XPath using Parent

parent goes one step up:

//h3[text()='Book']/parent::div

That returns the Book’s product-list container, and from there a query can reach the price and link that sit next to the known heading, which is the everyday scraping move this axis exists for.

XPath using Descendant

descendant selects children at any depth. Note the singular, since descendants:: is not an axis and fails with an invalid-expression error:

//div[@class="product-item"][1]/descendant::a

The first card’s buy link returns, even though it sits two levels below the card container.

XPath using Ancestor

ancestor is the singular-only counterpart going up, and it returns every container above the node, not just one:

//h3[text()='Pen']/ancestor::div

That returns both the product-list and the product-item divs. Add a condition, like ancestor::div[@class="product-item"], to land on the container you mean.

Conclusion and Takeaways

XPath matches on text, walks up to parents, and reaches siblings in either direction, which is the ground a CSS selector leaves uncovered. The axes all follow one pattern, anchor on what you can identify, then walk to what you need, and the parenthesized index and the singular axis names are the two syntax details that cost the most debugging time. The same expressions run unchanged in DevTools’ $x(), in Selenium through By.XPATH, and in lxml, so a query tested in the console is already the query your Python scraper ships with.