How to Parse XML Files for Data Extraction

Key Takeaways
Understanding XML Files
XML Structure
Fundamental XML Navigation
Advanced XML Navigation
XML Parsing Clients
XML in Python
XML in PHP
XML in JavaScript
XML in Other Languages
XML Parsing Project
FAQ
What is parsing in XML?
How to parse XML in Python?
What’s the difference between XML and HTML?
Summary

XML files are essential data formats used for structuring and organizing data in hierarchical layouts. They serve as critical components for data exchange between servers and clients, making them invaluable sources for information extraction. But how can we effectively parse XML files to extract meaningful data?

This comprehensive guide explores effective techniques for parsing XML files for data extraction purposes. We’ll begin by examining XML fundamentals, its structural components, and navigation methods. Next, we’ll explore parsing approaches using query languages and built-in tools. Finally, we’ll demonstrate practical implementation through a Python example.

Key Takeaways

Master xml parser techniques using XPath expressions, CSS selectors, and Python ElementTree for structured data extraction from XML documents.

Use XPath expressions for precise XML navigation and complex element selection with axes and functions
Parse XML with CSS selectors for simpler element targeting using familiar web development syntax
Handle XML parsing with language-native tools like lxml in Python for performance and flexibility
Navigate XML tree structures using parent-child relationships and hierarchical element access
Implement proper namespace handling for XML documents with multiple schema definitions
Extract structured data from XML using query languages for automated data processing workflows

Understanding XML Files

XML (eXtensible Markup Language) is a versatile markup language created for generating files that maintain readability for both humans and machines. These documents facilitate data exchange and transmission between various systems, including servers, websites and databases.

Unlike HTML, XML documents don’t rely on predetermined tags. Instead, they offer high customization capabilities and permit custom tag definitions, resulting in enhanced flexibility and improved readability. To grasp this concept better, let’s examine XML structure fundamentals.

XML Structure

XML documents follow hierarchical structures where elements nest within one another. These elements serve as the fundamental building blocks of XML files. Consider this simple XML example:

<?xml version="1.0" encoding="UTF-8"?>
<electronic_devices>
    <device>
        <name>Core I5 Laptop</name>
        <brand>HP</brand>
        <price>400

lt;/price> </device> </electronic_devices>

The <electronic_devices> serves as the root element, while <device> represents a parent containing <name>, <brand> and <price> as child elements.

When navigating XML files programmatically, they’re represented as tree-based structures:

XML elements can incorporate attributes that provide additional context and details. These attributes enable more precise and efficient element selection and parsing:

<device category="laptops">
    <name>Core I5 Laptop</name>
    <!-- more elements-->
</device>

Now that we understand XML files and their structure, let’s explore XML parsing specifics.

For parsing XML files, several options exist:

Query languages, including XPath and CSS
Native XML parsers, such as ElementTree in Python

However, native XML parsers can be restrictive and less efficient. Query languages offer superior capabilities due to their advanced features. Additionally, XPath and CSS selectors function with HTML documents as well.

We’ll examine XML parsing with XPath and CSS selectors thoroughly, then briefly cover built-in XML parsing modules. Before diving into details, let’s review XPath and CSS selector fundamentals. The following table shows commonly used XPath and CSS expressions:

XPath expression	CSS expression	description
//node	node	selects any descendant (child, grandchild, gran-grandchild etc.) that matches the node name
/node	>node	selects a direct child that matches the node name.
[@attribute=]	[attribute=]	selects a node by matching the attribute name or value.
/@attribute_name	::attr(attribute_name)	selects an attribute value of a node e.g. href attribute of an a node
/text()	:text	selects a text value of a node.
/	>	sequence multiple selectors together.

These expressions are commonly shared between XPath and CSS selectors, though each has slight differences. For comprehensive details about these expressions, refer to our previous articles and XPath and CSS cheat sheets.

When parsing XML, understanding element navigation is crucial. For instance, these selectors can target specific elements based on their parent relationships:

<products> <product> <id>1</id> <name type="title">Box of Chocolate Candy</name> <price>24.99</price> <rate value="4.5 out of 5">Product rate</rate> </product> </products>

Here, we extract the product name by selecting the products element and navigating through its children to reach the desired element.

We accessed the element by explicitly declaring its path. However, this method can prove inefficient when working with complex XML files. Instead, shortening the path based on element context works better. For example, the product name contains a type attribute of “title”. We can utilize this attribute for direct element selection:

<products> <product> <id>1</id> <name type="title">Box of Chocolate Candy</name> <price>24.99</price> <rate value="4.5 out of 5">Product rate</rate> </product> </products>

This modification eliminates lengthy selector expressions while making them more precise and reliable.

The previous expression can select elements by matching against their attributes. Alternatively, we can reverse this logic and retrieve the element’s attribute value directly:

<products> <product> <id>1</id> <name type="title">Box of Chocolate Candy</name> <price>24.99</price> <rate value="4.5 out of 5">Product rate</rate> </product> </products>

Here, we extract the rate value by selecting a specific element value. We can enhance this further by searching for attributes containing partial values:

<products> <product> <id>1</id> <name type="title">Box of Chocolate Candy</name> <price>24.99</price> <rate value="4.5 out of 5">Product rate</rate> </product> </products>

We’ve explored XML parsing using fundamental XPath and CSS selector methods. Let’s examine more sophisticated approaches.

With less descriptive XML files, targeting specific elements becomes challenging. Therefore, we may need to parse XML elements using indexes or positions. For instance, this XML code contains four identical elements. Here’s how we can access specific ones by matching their position:

<steps> <step>intro step</step> <step>first step</step> <step>second second</step> <step>outro step</step> </steps>

Here, we use position in XPath and nth in CSS to select the 2nd and 3rd elements from the XML file. We can also target these elements by matching or excluding based on element text:

<steps> <step>intro step</step> <step>first step</step> <step>second second</step> <step>outro step</step> </steps>

Here, we select elements that don’t contain specific text. Note that CSS selectors cannot select elements using text values. XPath selectors excel in this scenario.

When handling complex XML files, selecting specific elements becomes challenging as element breadth increases. However, we can overcome this by selecting elements based on their children’s attributes or values:

<product> <variant> <price>99

lt;/price> <rate type="product A">4.5</rate> </variant> <variant> <price>120

lt;/price> <rate type="product B">3.5</rate> </variant> </product>

Here, we select a specific variant based on its rate type and then target its price child.

We’ve covered various XPath and CSS selector examples for XML parsing. However, XML files are designed for machine and human readability, so encountering most previous challenges is unlikely.

XML Parsing Clients

We’ve explored parsing XML using query languages like XPath and CSS. However, most programming languages include built-in XML clients. Let’s examine each language’s XML client options.

XML in Python

For parsing XML files in Python, various packages are available. However, ElementTree remains the most popular choice. ElementTree enables XML file searching through their elements and supports most XPath syntax.

When using ElementTree, ensuring your XML is well-formed is crucial. Otherwise, you might encounter an xml parse error, particularly with missing tags or malformed elements.

Here’s a basic overview of XML parsing using ElementTree:

import xml.etree.ElementTree as ET

xml_data = '''
<root>
  <products>
    <product>
      <name attribute="product_name">Dark Red Energy Potion</name>
      <price attribute="product_price">$4.99</price>
      <rate attribute="product_rate">4.7</rate>
      <description attribute="product_description">Bring out the best in your gaming performance.</description>
    </product>
  </products>
</root>
'''

# Parse the XML data
root = ET.fromstring(xml_data)

# Iterate over elements
for parent in root:
    print(f"Parent tag: {parent.tag}")
    for child in parent:
        print(f"Child tag: {child.tag}")
        for grandchild in child:
            print(f"Grandchild tag: {grandchild.tag}, Attribute: {grandchild.attrib}")

# Access elements
product = root.find(".//product")
name = product.find(".//name").text
price = product.find(".//price").text
rate = product.find(".//rate").text 
description = product.find(".//description").text 

print(f"Name: {name}, Price: {price}, Rate: {rate}, Description: {description}")

Additional Python XML parsing tools include:

Minidom – A built-in library that converts XML files into a DOM representation.
xml.sax – A built-in SAX library for parsing XML in an event-driven approach.
lxml – A third-party library for processing XML and HTML files in high performance.

XML in PHP

The most popular XML processing tool in PHP is SimpleXML:

<?php
$xml_data = '
<root>
  <products>
    <product>
      <name attribute="product_name">Dark Red Energy Potion</name>
      <price attribute="product_price">$4.99</price>
      <rate attribute="product_rate">4.7</rate>
      <description attribute="product_description">Bring out the best in your gaming performance.</description>
    </product>
  </products>
</root>
';

// Parse the XML data
$xml = simplexml_load_string($xml_data);

// Iterate over elements
foreach ($xml->children() as $parent) {
    echo "Parent tag: {$parent->getName()}\n";
    foreach ($parent->children() as $child) {
        echo "Child tag: {$child->getName()}\n";
        foreach ($child->attributes() as $attr => $value) {
            echo "Attribute: $attr = $value\n";
        }
        foreach ($child->children() as $grandchild) {
            echo "Grandchild tag: {$grandchild->getName()}, Attribute: {$grandchild->attributes()['attribute']}\n";
        }
    }
}

// Access elements
$product = $xml->xpath('//product')[0];
$name = (string)$product->name;
$price = (string)$product->price;
$rate = (string)$product->rate;
$description = (string)$product->description;

echo "Name: $name, Price: $price, Rate: $rate, Description: $description\n";
?>

Other PHP XML parsing tools include:

DOMDocument – Converts HTML and XPath files into a DOM representation.
XMLReader – A memory-efficient tool for processing extensive XML files, it reads XML files in a streaming approach.

XML in JavaScript

JavaScript natively supports processing XML through the DOM in the browser. The most popular implementation of this interface is the DOMParser:

const xmlData = `
<root>
  <products>
    <product>
      <name attribute="product_name">Dark Red Energy Potion</name>
      <price attribute="product_price">$4.99</price>
      <rate attribute="product_rate">4.7</rate>
      <description attribute="product_description">Bring out the best in your gaming performance.</description>
    </product>
  </products>
</root>
`;

// Parse the XML data
const parser = new DOMParser();
const xmlDoc = parser.parseFromString(xmlData, 'text/xml');

// Access elements
const product = xmlDoc.getElementsByTagName('product')[0];
const name = product.getElementsByTagName('name')[0].textContent;
const price = product.getElementsByTagName('price')[0].textContent;
const rate = product.getElementsByTagName('rate')[0].textContent;
const description = product.getElementsByTagName('description')[0].textContent;

console.log(`Name: ${name}, Price: ${price}, Rate: ${rate}, Description: ${description}`);

Additional tool recommendations:

xml2js – A module that converts XML into JavaScript objects.
xmldom – An implementation of the DOMParser for Node.js environments.

XML in Other Languages

XML files are commonly used for data exchange, so most programming languages include built-in XML clients for processing them. XPath clients can also handle XML parsing and are available in nearly all programming languages, either as built-in features or community packages such as:

nokogiri and rexml in Ruby.
encoding/xml in Golang
libxml2 in C.
jdom in Java.

XML Parsing Project

In this section, we’ll create a Python XML parser to extract XML data from a real target website. We’ll be scraping real estate property links from this Realtor.com page. It contains the latest added properties on the website. The XML data on this page appears like this:

<?xml version="1.0" encoding="utf-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
  <channel>
    <title>Price Changed</title>
    <link>https://www.realtor.com</link>
    <description/>
    <atom:link href="https://pubsubhubbub.appspot.com/" rel="hub"/>
    <atom:link href="https://www.realtor.com/realestateandhomes-detail/sitemap-rss-price/rss-price-ca.xml" rel="self"/>
    <item>
      <link>https://www.realtor.com/realestateandhomes-detail/275-Howell-Ave_Red-Bluff_CA_96080_M27031-58587</link>
      <pubDate>Fri, 05 Jan 2024 13:43:02</pubDate>
    </item>
    <item>
      <link>https://www.realtor.com/realestateandhomes-detail/2575-Thunder-Mountain-Rd_Upland_CA_91784_M19026-44599</link>
      <pubDate>Fri, 05 Jan 2024 13:47:32</pubDate>
    </item>
    <item>
      <link>https://www.realtor.com/realestateandhomes-detail/20631-W-Chestnut-Cir_Porter-Ranch_CA_91326_M90130-33610</link>
      <pubDate>Fri, 05 Jan 2024 13:47:34</pubDate>
    </item>
  </channel>
</rss>

To scrape this data, we’ll request the page URL and then parse the XML content using the XPath selector:

Python

Webparsers

import json
from httpx import Client
from parsel import Selector

client = Client(
    # enable http2
    http2=True,
    # add basic browser like headers to prevent being blocked
    headers={
        "accept-language": "en-US,en;q=0.9",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
        "accept-encoding": "gzip, deflate, br",
    },
)

# send a request
response = client.get("https://www.realtor.com/realestateandhomes-detail/sitemap-rss-price/rss-price-ca.xml")
# get the XML content from the response body
body = response.content
# convert the XML to a Pasrsel selector
selector = Selector(response.text, type="xml")
results = {}
# iterate over the items
for item in selector.xpath("//item"):
    url = item.xpath("./link/text()").get()
    pub_date = item.xpath("./pubDate/text()").get()
    results[url] = pub_date

# save the results into a JSON file
with open("data.json", "w", encoding="utf-8") as file:
    json.dump(results, file, indent=2, ensure_ascii=False)

Here, we request the page URL to retrieve the XML content. Then, we convert the received data into a Parsel XML selector. Finally, we parse the data to extract the URL and date of each item and save the data into a JSON file. Here’s a sample output of our results:

{
  "https://www.realtor.com/realestateandhomes-detail/231-S-Laxore-St_Anaheim_CA_92804_M14771-41700": "Fri, 05 Jan 2024 14:39:35",
  "https://www.realtor.com/realestateandhomes-detail/60815-Burnt-Valley-Rd_Anza_CA_92539_M21181-92033": "Fri, 05 Jan 2024 14:39:43",
  "https://www.realtor.com/realestateandhomes-detail/463-Buckeye-St_Vacaville_CA_95688_M24868-08760": "Fri, 05 Jan 2024 14:50:33",
  "https://www.realtor.com/realestateandhomes-detail/61-Arthur-Dr_Santa-Rosa_CA_95403_M29889-89374": "Fri, 05 Jan 2024 14:57:14",
  "https://www.realtor.com/realestateandhomes-detail/29413-Fenders-Ferry-Rd_Round-Mountain_CA_96084_M29662-29330": "Fri, 05 Jan 2024 15:00:47"
}

FAQ

To conclude this guide, let’s examine some frequently asked questions about XML parsing.

What is parsing in XML?

Parsing in XML means searching through XML elements to extract their data. This can be accomplished by matching against elements’ tags, attributes, or values.

How to parse XML in Python?

You can parse XML files in Python using tools such as ElementTree, lxml, or query languages like XPath. Be sure to handle potential issues like an xml parse error using proper error handling.

What’s the difference between XML and HTML?

HTML is an XML-based markup language used for creating web pages so it’s a more restrictive version of XML. Fortunately, both markups are very similar and can often be parsed using the same tools.

Summary

In this article, we explored XML files – markup language files used for exchanging data in a readable format.

We have examined how to use query languages such as XPath and CSS to parse XML, which enables precise element selection with minimal expressions. We also reviewed available methods for parsing XML in different programming languages.

Finally, we demonstrated a practical example project creating a Python XML parser.

How to Parse and Extract Data from XML Files: A Complete Guide

Table of Contents

Key Takeaways

Understanding XML Files

XML Structure

Fundamental XML Navigation

Advanced XML Navigation

XML Parsing Clients

XML in Python

XML in PHP

XML in JavaScript

XML in Other Languages

XML Parsing Project

Python

Webparsers

FAQ

What is parsing in XML?

How to parse XML in Python?

What’s the difference between XML and HTML?

Summary

You might also like

How to Scrape X.com (Twitter) in 2026

How to Use Playwright Stealth to Prevent Bot Detection When Web Scraping

Ultimate Guide on Bypassing Amazon CAPTCHA in 2025: Expert Tips and Techniques

NOT SURE WHERE TO START?

Let us help you to achieve your business goals