XPath Cheat Sheet for Web Scraping – Guide & Examples

XML Path Language (XPath) is a query language and a major component of the XSLT standard. It employs a path-like syntax (referred to as path expressions) to identify and navigate nodes within XML and XML-like documents.

In web scraping applications, we can leverage XPath to locate and select elements from the DOM tree of virtually any HTML document, enabling us to build more robust parsers in our scripts.

By the conclusion of this guide, you’ll possess a comprehensive understanding of XPath expressions and their implementation in your scripts for scraping complex websites.

Understanding XPath Syntax

Creating XPath expressions is straightforward because it employs a structure familiar to most of us. You can visualize these path expressions similar to those used in standard file systems.

There’s a root directory, which contains various subdirectories that may also house additional folders. XPath leverages the relationships between these elements to traverse the tree structure and locate our target elements.

For instance, we can utilize the expression //div to select all div elements or write //div/p to target all paragraphs within the divs. This works due to the hierarchical nature of HTML.

Using XPath to Find Elements With Chrome Dev Tools

Let’s use a practical example to illustrate this concept. Navigate to https://quotes.toscrape.com/ and inspect the page.

Now we’ll be able to examine the website’s HTML structure and select an element using our XPath expressions. To scrape all quotes displayed on the page, simply press cmd + f to initiate a search and write our expression.

Note: This is an excellent exercise to test your expressions before investing time in your code editor and without putting stress on the site’s server.

Upon closer examination, we can observe that all quotes are contained within a div with the class “quote”, with the actual text inside a span element with the class “text”. Let’s follow this structure to construct our path:

XPath: //div[@class='quote']/span[@class='text']

This highlighted the first matching element and indicates it’s the first of 10 elements, which perfectly corresponds to the number of quotes on the page.

Note: The expression //span[@class='text'] would also function correctly since there’s only one span using that class. We aim to be as descriptive as possible because, in most scenarios, we’ll be using XPath on websites with more complex structures.

Notice how we’re using the elements’ attributes to locate them? XPath enables us to navigate in any direction and in multiple ways through the node tree. We can target classes, IDs, and the relationships between elements.

For the previous example, we can write our path as: //div[@class='quote']/span[1] and still locate the element. This expression translates to finding all divs with the class “quote” and selecting the first span element.

To summarize what we’ve covered so far, here’s the XPath syntax structure:

• Tagname – the name of the HTML element itself (divs, H1s, etc.)
• Attribute – IDs, classes, and any other properties of the HTML element we’re targeting
• Value – the value stored in the attribute of the HTML element

If you’re still struggling with this syntax, an excellent starting point is understanding data parsing fundamentals and their mechanics. In-depth knowledge of the DOM and its structure will make XPath concepts much clearer.

When to Use XPath vs. CSS for Web Scraping

If you’ve explored Beautiful Soup tutorials or Cheerio guides, you’ve probably noticed that CSS is frequently used in most projects. This preference is primarily due to practical considerations.

In real-world projects, situations become more complex, and understanding both approaches provides you with additional tools to tackle any challenge.

Let’s examine the differences between XPath and CSS selectors to understand when each should be employed.

Advantages of CSS for Web Scraping

CSS selectors are generally easier to write and read compared to XPath selectors, making them more accessible for both learning and implementation.

For comparison, here’s how we would select a paragraph with the class “easy”:

• XPath: //p[@class="easy"]
• CSS: p.easy

Another consideration is that when working with websites structured with unique IDs and distinct classes, CSS becomes the optimal choice because element selection based on CSS selectors is more reliable.

One DOM change can break our XPath, making our script vulnerable. However, classes and IDs rarely change, allowing you to select elements regardless of positional alterations.

While this might be somewhat opinionated, we consider CSS our primary option for projects, only switching to XPath when we cannot find an efficient CSS solution.

Recommended: The Ultimate CSS Selectors Cheat Sheet for Web Scraping

Advantages of XPath for Web Scraping

Unlike CSS, XPath can traverse the DOM tree both upward and downward, providing greater flexibility when working with less structured websites. This creates numerous opportunities for DOM interaction that CSS doesn’t offer.

Consider needing to select a specific parent div from a document containing 15 different divs without any class, ID, or attribute. CSS cannot be used effectively because there are no suitable targets to handle.

However, with XPath, we can target a child element of the desired div and navigate upward from there.

Using //span[@class="text"]/.., we create a path to find the span element with class “text” and then move to that specific span’s parent element, effectively navigating up the DOM.

Another excellent application of XPath selectors/expressions is finding elements by matching their text content – something impossible with CSS – using the contains function.

XPath: //*[contains(text(), 'world')]

In this example, our XPath expression matches two elements because both the quote and the tag contain the word “world”. While we may not use this function frequently in web scraping, it’s valuable in specific situations.

For more information about the differences between these approaches, we recommend Exadel’s guide on selecting selectors for automation.

Although not directly related to web scraping, there’s considerable value in learning automation concepts.

XPath is a powerful language essential in many scenarios, so let’s examine some common expressions useful for web scraping.

XPath Cheat Sheet: Common Expressions for Web Scraping

For those involved in web scraping, here’s a practical cheat sheet for daily use. Bookmark it and enjoy!

Note: You can test every expression in the “example” column on Quotes to Scrape for additional clarity and to see your selections. Except for the ID example because the website doesn’t use IDs.

You want to:	Syntax	Example
Pick elements from anywhere in the DOM with a specific HTML tag	//tagName	//span
Pick an element by ID	//tagName[@id="idValue"]	//div[@id="main-product"]
Pick an element by class	//tagName[@class="classValue"]	//div[@class="quote"]
Pick the child of an element	//parentName/childName	//div/span
Pick the first child of an element	//parentName/childName[1]	//div[@class="quote"]/span[1]
Pick the parent of an element	//childName/..	//span[@class="text"]/..
Get the anchor of an element	//tagName/@href	//a/@href
Pick an element by matching its text	//tagName[text()="exactText"]	//span[text()="Login"]
Pick an element that partially matches some text	//tagName[contains(text(), "someText")]	//span[contains(text(), "by")]
Picking the last child of an element or the last element in a list	//parentName/childName[last()]	//div[@class="quote"]/span[last()]

These are likely the most common XPath expressions you’ll use to select elements from HTML documents. However, these aren’t the only ones available, and we encourage continued learning.

For example, you can also select elements that don’t contain certain text using the expression //tagName[not(contains(text(), "someText"))]. This could be useful when websites add text to elements based on variables, like adding “out of stock” to product titles within category pages.

We can also implement OR logic when working with classes that change based on variables using //tagName[@class="class1" or @class="class2"]. This tells our scraper to select elements that have either class.

In a previous project involving stock data scraping, the “price percentage change” class name varied depending on whether the price was increasing or decreasing. Because the change was consistent, we could easily implement OR logic with XPath and make our scraper extract values regardless of which class the element used.

XPath: //span[@class="instrument-price_change-percent__19cas ml-2.5 text-positive-main" or @class="instrument-price_change-percent__19cas ml-2.5 text-negative-main"]

XPath Web Scraper Example

Before concluding, we want to share a script written in Puppeteer to demonstrate these XPath selectors in action.

Note: Technologies like Cheerio or Beautiful Soup don’t work well – and in some cases not at all – with XPath, so we recommend using tools like Scrapy for Python and Puppeteer for JavaScript when XPath is needed. These are more complex tools initially, but you’ll become proficient quickly.

Create a new folder called “xpathproject”, open it in VSCode (or your preferred editor), initialize a new Node.js project using npm init -y, and install puppeteer with npm install puppeteer.

Next, create a new file with any name you prefer (we named it index.js for simplicity) and paste the following code:

const puppeteer = require('puppeteer'); 


scrapedText = []; 

(async () => { const browser = await puppeteer.launch({headless: false}); 

const page = await browser.newPage(); 

await page.goto('https://quotes.toscrape.com/'); 

await page.waitForXPath('//div[@class="quote"]/span[1]'); 

let elements = await page.$x('//div[@class="quote"]/span[1]'); 

const elementText = await page.evaluate((...elements) => { return elements.map(el => el.textContent); }, ...elements) 

scrapedText.push({ 'Results': elementText }) 

console.log(scrapedText); 

await browser.close(); })();

If you lack experience with Puppeteer, check our guide on building web scrapers using Cheerio and Puppeteer. However, if you read this script carefully, you’ll notice how descriptive it is.

The best part is that you can use any XPath example from the cheat sheet table and replace the expressions in the script, and it will extract the text from the elements it finds.

It’s important to note that this web scraper is designed for extracting text from multiple elements, so it might not work for simply taking the page title, for example.

Try different combinations and experiment with the script. You’ll soon master XPath. Until next time, happy scraping!