Skip to main content

Webparsers.com

Webparsers Homepage
Login
Menu
Menu
Login

XPath Cheat Sheet for Web Scraping

10 min read

Table of Contents

XPath Cheat Sheet for Web Scraping

XPath (XML Path Language) is a query language for navigating document trees. Originally designed for XML, it works equally well on HTML — which is why every major web scraping framework supports it alongside CSS selectors. XPath's key advantage over CSS is directionality: CSS selectors can only traverse the DOM downward (from parent to child), while XPath can move in any direction — including upward to parent and sibling elements, and sideways to preceding and following nodes. It can also select elements by their text content, which CSS cannot do.

This reference covers the XPath expressions most useful for web scraping — with a syntax cheat sheet, Python code examples using Scrapy and lxml, a comparison with CSS selectors, and the scenarios where XPath is the right choice. Webparsers builds data collection pipelines that handle complex extraction logic across structured and unstructured HTML targets — see our API Marketplace for available endpoints.

Talk to a Data Engineer

XPath Syntax Basics

An XPath expression describes a path through the document tree, similar to a filesystem path. The key syntax elements:

  • // — Select matching nodes anywhere in the document (most common in scraping)
  • / — Select a direct child of the current node
  • . — The current node
  • .. — The parent of the current node
  • @ — Select an attribute: @class, @href, @id
  • [ ] — Predicate: filters results by condition, position, or attribute value
  • * — Wildcard: matches any element tag name

XPath Cheat Sheet

Goal XPath expression Notes
All elements of a tag //div Selects all <div> elements anywhere in the document
Select by exact class //div[@class="product-card"] Matches only when the class attribute is exactly that value — fails on multiple classes
Class contains word //div[contains(@class,"product")] Matches elements where the class attribute contains "product" as a substring
Select by ID //div[@id="main-content"] ID attributes are unique per page — equivalent to CSS #main-content
Direct child //div[@class="card"]/h2 The h2 that is a direct child of the matched div
Any descendant //div[@class="card"]//span Any span at any depth inside the matched div
Parent element //span[@class="price"]/.. The parent of the matched span — not possible with CSS selectors
First child //ul/li[1] XPath position indices start at 1, not 0
Last child //ul/li[last()] Selects the last li in each ul
Text content //h2/text() The direct text node of the element (excludes child element text)
All text in subtree //div[@class="desc"]//text() All text nodes at any depth — join with " ".join() in Python
Match by text content //td[text()="In Stock"] Selects elements whose text content is exactly the value — not possible with CSS
Text contains substring //span[contains(text(),"Save")] Matches elements whose text contains "Save" anywhere — useful for promotional labels
Attribute value //a/@href Selects the href attribute value directly from all a elements
Attribute starts with //a[starts-with(@href,"/product")] Equivalent to CSS a[href^="/product"]
OR on attribute values //span[@class="price-up" or @class="price-down"] Matches elements with either class — useful when class names change with state
Exclude by text //span[not(contains(text(),"Out of stock"))] Negation — selects elements whose text does not contain the string
Following sibling //h3/following-sibling::p[1] The first p element after the matched h3 at the same level
Preceding sibling //p/preceding-sibling::h3[1] The nearest h3 before the matched p
Wildcard tag //*[@data-price] Any element with a data-price attribute, regardless of tag type

XPath in Python: Scrapy and lxml

Scrapy

def parse(self, response):
    for item in response.xpath('//div[contains(@class,"product-card")]'):
        yield {
            # Extract direct text node
            "name": item.xpath('.//h2/text()').get(),
            # Extract attribute value
            "url": item.xpath('.//a/@href').get(),
            # Match by text content — selects sibling price after label
            "price": item.xpath('.//span[contains(@class,"price")]/text()').get(),
            # Collect all text nodes in a description div
            "description": " ".join(
                item.xpath('.//div[@class="desc"]//text()').getall()
            ).strip(),
        }

    # Pagination via attribute
    next_page = response.xpath('//a[@rel="next"]/@href').get()
    if next_page:
        yield response.follow(next_page, self.parse)

In Scrapy, XPath calls on a sub-selector (inside a loop) use . at the start of the expression to scope to the current node rather than the document root. Omitting the leading . re-queries the entire document, returning all matching elements rather than those within the current item.

lxml (Python)

from lxml import html
import requests

response = requests.get(url, headers=headers, proxies=proxies)
tree = html.fromstring(response.content)

# Select elements and extract text
titles = tree.xpath('//h2[contains(@class,"product-title")]/text()')

# Navigate to parent
parent_divs = tree.xpath('//span[@class="price"]/..')

# Select by text match
in_stock = tree.xpath('//td[text()="In Stock"]/../td[1]/text()')

# OR logic for dynamic class names
prices = tree.xpath(
    '//span[@class="price-positive" or @class="price-negative"]/text()'
)

lxml's .xpath() returns Python lists directly. For attribute values, the expression returns strings; for element nodes, it returns lxml element objects from which you call .text or .get("attribute").

XPath vs CSS Selectors: When to Use Each

Requirement CSS selector XPath
Select by class or ID .product-title — cleaner //*[contains(@class,"product-title")] — verbose
Navigate to parent element Not possible //span[@class="price"]/..
Select by text content Not possible //td[text()="In Stock"]
Dynamic class name (state-dependent) Fragile — each class variant needs its own rule @class="a" or @class="b" — single expression
Attribute starts with / ends with a[href^="/product"] — cleaner a[starts-with(@href,"/product")] — equivalent
Nth child li:nth-child(2) //li[2] (note: XPath is 1-indexed)
Following/preceding sibling h3 + p (adjacent only) //h3/following-sibling::p[1] — more flexible
Readability Generally more concise and readable More verbose; harder to read at a glance

The practical rule: start with CSS selectors. Switch to XPath when CSS cannot express the required selection — particularly for parent traversal, text-content matching, and OR conditions on dynamic class names.

Common XPath Patterns for Scraping Unstructured Pages

Extracting Data When the Target Has No Class or ID

Older or less-maintained sites often render data in plain HTML without class names. When neither CSS class nor ID is available, XPath structural expressions navigate by position and tag type: 

# Select the third td in each table row (e.g., a price column)
prices = tree.xpath('//table[@id="results"]//tr/td[3]/text()')

# Select a label's adjacent value cell in a definition table
value = tree.xpath('//td[text()="Price"]/following-sibling::td[1]/text()')

Handling Dynamic Class Names

Some sites apply class names that include state information — price--positive vs price--negative, or btn-active vs btn-inactive. The contains() function matches on a stable substring: 

# Match both variants using contains
changes = tree.xpath('//span[contains(@class,"price--")]/text()')

# Match using OR for exactly two known variants
changes = tree.xpath(
    '//span[@class="price--positive" or @class="price--negative"]/text()'
)

Combining with Proxy-Based Collection

XPath handles extraction once a valid HTML response is in hand. For protected targets, the collection layer must deliver that response cleanly — which requires residential proxies for IP-based filtering and browser automation for JavaScript-rendered pages. See our article on proxy management for how proxy rotation is configured for scraping pipelines.

How Webparsers Applies XPath in Production Data Pipelines

  1. We choose XPath or CSS per target based on the DOM structure. For pages with semantic, stable class names, CSS selectors are used for readability and maintainability. XPath is applied when: element selection requires parent traversal, text-content matching is needed to locate unlabelled data fields, or class names are dynamic and require OR or contains() logic. The choice is made per target during the extraction design phase, not applied uniformly across all sources. See our API Docs and API Marketplace.
  2. We write extraction logic with per-field fallback expressions. Production pages often have structural variants — an out-of-stock product page omits the price field; a sponsored result has a different DOM structure than an organic result. We write primary and fallback selectors for fields likely to be absent or differently structured, so extraction handles page variants without per-page configuration. Fallback returns None rather than raising an error.
  3. We handle JavaScript-rendered pages with browser-based extraction. Playwright exposes the fully rendered DOM to XPath queries via page.locator() with XPath arguments or via page.evaluate() with custom JavaScript. For targets where data is injected by JavaScript, we execute XPath against the post-render DOM rather than the pre-render HTML response. See our article on headless browsers for scraping.
  4. We normalise extracted values before delivery. XPath text node extraction returns raw strings as they appear on the page — including currency symbols, trailing whitespace, and encoding artefacts. We apply type-specific normalisation: stripping and parsing price strings to numeric values with currency codes, converting date strings to ISO 8601, joining multi-node text extractions into clean strings. See our article on data normalization and enrichment.
  5. We monitor extraction output to detect structural failures silently. When a site change causes an XPath expression to stop matching — class names are renamed, layout is restructured — the extractor returns None for that field without raising an error. We monitor field non-null rates per collection run and flag anomalous drops as extraction failures requiring selector review, before they propagate to delivered data as empty records. See our article on data delivery and integration.

Discuss Your Data Collection Requirements

Frequently Asked Questions

What is XPath in web scraping?

XPath (XML Path Language) is a query language for navigating and selecting nodes in an HTML or XML document tree. In web scraping, XPath expressions target elements by tag name, attribute value, text content, or position relative to other elements. It is supported in Scrapy (response.xpath()), lxml (Python), and Playwright. XPath is preferred when you need to traverse the DOM upward to a parent element or select elements by their text content — both of which CSS selectors cannot do.

When should I use XPath instead of CSS selectors?

Use XPath when: the target element has no useful class or ID and can only be identified by text content (contains(text(),'value')); you need to navigate to a parent element (/..); elements use dynamic class names that change between page states (XPath OR logic: @class='class-a' or @class='class-b'); or you need to select by partial attribute match. Use CSS selectors for everything else — they are more concise and readable for structural targeting.

What is the XPath syntax for selecting by class?

For an exact class match: //div[@class="product-card"]. For elements with multiple classes, use contains(): //span[contains(@class,"price")]. This matches the class attribute regardless of other classes present, equivalent to the CSS .price selector. The exact match [@class="price"] would fail on elements with class="price sale-price".

How do I extract text with XPath?

Use the text() node function: //h2[@class="product-title"]/text() returns the direct text node of the element. In Scrapy, chain .get() or .getall(): response.xpath('//h2/text()').get(). For elements with mixed content (text interspersed with child elements), use .//text() to collect all text nodes in the subtree and join them in Python.

What is the difference between // and / in XPath?

A single slash (/) selects a direct child of the current node; /html/body/div selects only a div that is an immediate child of body. Double slash (//) selects matching nodes anywhere in the document or subtree, regardless of depth: //div selects all div elements anywhere. In web scraping, // is used almost exclusively because target elements are rarely at a known fixed depth from the document root.