Tabular data represents one of the most valuable sources of information on the web. These structures can store massive amounts of useful data while maintaining an easy-to-read format, making them invaluable resources for data-driven projects.
Whether extracting football statistics or gathering stock market information, Python provides powerful tools to quickly access, parse and extract data from HTML tables using libraries like Requests and Beautiful Soup.
We also have a special surprise for you at the end, so keep reading!
Understanding HTML Table’s Structure
Visually, an HTML table consists of rows and columns that display information in a structured tabular format. For this tutorial, we’ll be scraping the table shown above.
To successfully extract data from this table, we need to examine its underlying HTML structure.
HTML tables are typically constructed using these essential HTML tags:
<table>: Marks the beginning of an HTML table<th>or<thead>: Defines table headers or heading rows<tbody>: Contains the main data section<tr>: Represents a table row<td>: Defines individual cells within the table
However, in real-world scenarios, not all developers follow these conventions when building tables, which can make some projects more challenging than others. Nevertheless, understanding these fundamentals is essential for choosing the right scraping approach.
Let’s navigate to the table’s URL (https://datatables.net/examples/styling/stripe.html) in our browser and inspect the page to examine its underlying structure.
This page provides an excellent opportunity to practice scraping tabular data with Python. There’s a clear <table> tag pair enclosing the table, and all relevant data is contained within the <tbody> tag. It displays only ten rows, which corresponds to the number of entries selected in the front-end interface.
A few important observations about this table: it contains a total of 57 entries we want to scrape, and there appear to be two methods to access the complete data. The first option involves clicking the drop-down menu and selecting “100” to display all entries at once.
The alternative approach requires clicking the next button to navigate through the pagination.
Which approach should we choose? Both solutions would add complexity to our script, so let’s first investigate where the data originates.
Since this is an HTML table, all data should be present in the HTML file itself without requiring AJAX injection. To verify this, right-click and select “View Page Source.” Next, copy several cell values and search for them in the source code.
We performed the same verification for multiple entries from different paginated sections, and indeed, all our target data exists in the source code even though the front-end doesn’t display it all at once.
With this information confirmed, we’re ready to begin coding!
Scraping HTML Tables Using Python’s Beautiful Soup
Since all the employee data we need is contained within the HTML file, we can use the Requests library to send HTTP requests and parse the response using Beautiful Soup.
Note: If you’re new to web scraping, we’ve created a comprehensive Python web scraping tutorial for beginners. While you can follow along without prior experience, starting with the fundamentals is always recommended.
1. Sending Our Initial Request
Let’s create a new directory for our project called python-html-table, then create a subfolder named bs4-table-scraper, and finally create a new file called python_table_scraper.py.
From the terminal, install the required packages with pip3 install requests beautifulsoup4 and import them into our project:
import requests
from bs4 import BeautifulSoupTo send an HTTP request using Requests, we need to define a URL and pass it through requests.get(), store the returned HTML in a response variable, and check response.status_code.
Note: If you’re completely new to Python, you can execute your code from the terminal using the command python3 python_table_scraper.py.
url = 'https://datatables.net/examples/styling/stripe.html'
response = requests.get(url)
print(response.status_code)A successful request will return a 200 status code. Any other status code indicates that your IP is being blocked by the website’s anti-scraping systems. One solution is adding custom headers to make your script appear more human-like, though this might not always be sufficient. Another approach is using a web scraping API to handle these complexities automatically.
2. Integrating Webparsers to Bypass Anti-Scraping Systems
Webparsers provides an elegant solution to circumvent almost any type of anti-scraping technique. It utilizes machine learning and extensive statistical analysis to determine optimal headers and IP combinations for accessing data, handling CAPTCHAs, and rotating your IP address between requests.
To get started, create a new account to receive 5000 free API calls and obtain your API key. From your account dashboard, copy the key value to construct the request URL.
http://api.scraperapi.com?api_key={Your_API_KEY}&url={TARGET_URL}Following this structure, replace the placeholders with your actual data and send the request again:
import requests
from bs4 import BeautifulSoup
url = 'http://api.scraperapi.com?api_key=51e43be283e4db2a5afbxxxxxxxxxxx&url=https://datatables.net/examples/styling/stripe.html'
response = requests.get(url)
print(response.status_code)Excellent, it’s working smoothly!
3. Building the Parser Using Beautiful Soup
Before extracting data, we need to convert the raw HTML into formatted or parsed data. We’ll store this parsed HTML in a soup object:
soup = BeautifulSoup(response.text, 'html.parser')From here, we can navigate the parse tree using HTML tags and their attributes.
Returning to the table on the page, we’ve already observed that the table is enclosed within <table> tags with the class “stripe dataTable,” which we can use to select the table.
table = soup.find('table', class_ = 'stripe')
print(table)Note: After testing, adding the second class (dataTable) didn’t return the element. The table’s class attribute is actually just “stripe.” You can also use id = 'example' as an alternative selector.
Now that we’ve captured the table, we can iterate through the rows and extract the desired data.
4. Iterating Through the HTML Table
Recalling the table’s structure, each row is represented by a <tr> element, containing <td> elements with data, all wrapped within a <tbody> tag pair.
To extract the data, we’ll create two for loops: one to grab the <tbody> section of the table (containing all rows) and another to store all rows in a usable variable:
for employee_data in table.find_all('tbody'):
rows = employee_data.find_all('tr')
print(rows)In the rows variable, we’ll store all <tr> elements found within the table’s body section. Following our logic, the next step involves storing each individual row in a single object and looping through them to extract the desired data.
Let’s test extracting the first employee’s name using our browser’s console with the .querySelectorAll() method. A useful feature of this method is that we can traverse deeper into the hierarchy using the greater than (>) symbol to define the parent element (left side) and the child element we want to grab (right side).
document.querySelectorAll('table.stripe > tbody > tr > td')[0]That works perfectly. As you can see, once we grab all <td> elements, they form a nodelist. Since we can’t rely on classes to grab each cell, we only need to know their index positions, with the first one (name) being at index 0.
We can write our code like this:
for row in rows:
name = row.find_all('td')[0].text
print(name)In simple terms, we’re processing each row individually, finding all cells inside, grabbing only the first one in the index (position 0), and finishing with the .text method to extract only the element’s text content, excluding the HTML markup we don’t need.
There they are—a list containing all employee names! For the remaining data, we follow the same logic:
position = row.find_all('td')[1].text
office = row.find_all('td')[2].text
age = row.find_all('td')[3].text
start_date = row.find_all('td')[4].text
salary = row.find_all('td')[5].textHowever, having all this data printed to our console isn’t particularly helpful. Instead, let’s store this data in a more useful format.
5. Storing Tabular Data in a JSON File
While we could easily create a CSV file for our data, that wouldn’t be the most manageable format if we want to build something new using the scraped data.
Here’s a project we completed recently explaining how to create a CSV file for storing scraped data.
The good news is that Python includes its own JSON module for working with JSON objects, so no additional installation is required—just import it.
import jsonBefore creating our JSON file, we need to convert all scraped data into a list. We’ll create an empty array outside our loop.
employee_list = []Then append data to it, with each loop iteration adding a new object to the array.
employee_list.append({
'Name': name,
'Position': position,
'Office': office,
'Age': age,
'Start date': start_date,
'salary': salary
})If we print(employee_list), here’s the result:
Still somewhat messy, but we have a collection of objects ready to be converted to JSON.
Note: As a test, we printed the length of employee_list and it returned 57, which matches the correct number of rows we scraped (rows now being objects within the array).
Converting a list to JSON requires just two lines of code:
with open('json_data', 'w') as json_file:
json.dump(employee_list, json_file, indent=2)First, we open a new file with the desired name (json_data) and ‘w’ since we want to write data to it. Next, we use the .dump() function to dump the data from the array (employee_list) and indent=2 so each object appears on its own line instead of everything being on one unreadable line.
6. Running the Script and Complete Code
If you’ve been following along, your codebase should look like this:
#dependencies
import requests
from bs4 import BeautifulSoup
import json
url = 'http://api.scraperapi.com?api_key=51e43be283e4db2a5afbxxxxxxxxxxx&url=https://datatables.net/examples/styling/stripe.html'
#empty array
employee_list = []
#requesting and parsing the HTML file
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
#selecting the table
table = soup.find('table', class_ = 'stripe')
#storing all rows into one variable
for employee_data in table.find_all('tbody'):
rows = employee_data.find_all('tr')
#looping through the HTML table to scrape the data
for row in rows:
name = row.find_all('td')[0].text
position = row.find_all('td')[1].text
office = row.find_all('td')[2].text
age = row.find_all('td')[3].text
start_date = row.find_all('td')[4].text
salary = row.find_all('td')[5].text
#sending scraped data to the empty array
employee_list.append({
'Name': name,
'Position': position,
'Office': office,
'Age': age,
'Start date': start_date,
'salary': salary
})
#importing the array to a JSON file
with open('employee_data', 'w') as json_file:
json.dump(employee_list, json_file, indent=2)Note: We added comments for context.
Here’s a preview of the first three objects from the JSON file:
Storing scraped data in JSON format allows us to repurpose the information for new applications.
Scrape HTML Tables with Complex Headers
Scraping data from HTML tables is generally straightforward, but what happens when you encounter tables with more complex structures, such as nested tables, rowspans, or colspans? In these cases, you might need to implement more sophisticated parsing logic.
Before we begin, let’s examine what the target table looks like:
As you can see, this table features a two-level header structure:
- The first row contains broader categories: “Name”, “Position”, and “Contact”.
- The second row further subdivides these categories.
Let’s scrape that table!
Setting Up the Scraping Environment
First, we must import the necessary libraries. Webparsers will help us handle any anti-scraping measures the website might implement, including managing headers and rotating IPs when necessary:
import requests
from bs4 import BeautifulSoup
import pandas as pd
api_key = 'YOUR_API_KEY'
url = 'https://datatables.net/examples/basic_init/complex_header.html'Creating the Scraping Function
Let’s create a function called scrape_complex_table that will handle the entire scraping process. This function will accept our URL as input and return a pandas DataFrame containing the structured table data:
def scrape_complex_table(url):
# Send a request to the webpage
payload = {'api_key': api_key, 'url': url}
response = requests.get('https://api.scraperapi.com', params=payload)
soup = BeautifulSoup(response.text, 'html.parser')This function uses requests.get to send a GET request to the scraping API, which fetches the target webpage. We then parse the HTML content using BeautifulSoup.
Locating the Target Table
We locate the table in the parsed HTML using its id attribute.
# Find the target table
table = soup.find('table', id='example')This line finds the first <table> element with id='example'.
Extracting and Combining Headers
We extract the first and second levels of headers and combine them to form a single list of column names.
# Extract and combine headers
headers_level1 = [th.text.strip() for th in table.select('thead tr:nth-of-type(1) th')]
headers_level2 = [th.text.strip() for th in table.select('thead tr:nth-of-type(2) th')]
combined_headers = []
for i, header in enumerate(headers_level1):
if header == 'Name':
combined_headers.append(header)
elif header == 'Position':
combined_headers.extend([f"{header} - {col}" for col in ['Title', 'Salary']])
elif header == 'Contact':
combined_headers.extend([f"{header} - {col}" for col in ['Office', 'Extn.', 'Email']])Here, we use CSS selectors to target the header rows. We then loop through the first-level headers and, depending on the header, append appropriate second-level headers to our combined_headers list.
Extracting Data from the Table Body
We extract the data from each row in the table body.
# Extract data from table body
rows = []
for row in table.select('tbody tr'):
cells = [cell.text.strip() for cell in row.find_all('td')]
rows.append(cells)This code loops through each <tr> in the <tbody>, extracts the text from each <td>, and stores the data in the rows list.
Creating the DataFrame
We’ll create a Pandas DataFrame using the combined headers and extracted data. This DataFrame organizes our data into a structured format with appropriate column names.
# Create DataFrame
df = pd.DataFrame(rows, columns=combined_headers)
return dfRunning the Scraper and Saving Data
We call the scrape_complex_table function, display the first few rows of the DataFrame, and save it to a CSV file.
result_df = scrape_complex_table(url)
# Display the first few rows of the result
print(result_df.head())
result_df.to_csv('complex_table_data.csv', index=False)
print("Data has been saved to 'complex_table_data.csv'")This will print the top rows of the DataFrame and save the entire dataset to a file named complex_table_data.csv.
Putting It All Together
Here’s what the complete code should look like after combining all the steps:
import requests
from bs4 import BeautifulSoup
import pandas as pd
api_key = 'your_api_key_here' # Replace with your actual ScraperAPI key
url = 'https://datatables.net/examples/basic_init/complex_header.html'
def scrape_complex_table(url):
payload = {'api_key': api_key, 'url': url}
response = requests.get('https://api.scraperapi.com', params=payload)
soup = BeautifulSoup(response.text, 'html.parser')
# Find the target table
table = soup.find('table', id='example')
# Extract and combine headers
headers_level1 = [th.text.strip() for th in table.select('thead tr:nth-of-type(1) th')]
headers_level2 = [th.text.strip() for th in table.select('thead tr:nth-of-type(2) th')]
combined_headers = []
for i, header in enumerate(headers_level1):
if header == 'Name':
combined_headers.append(header)
elif header == 'Position':
combined_headers.extend([f"{header} - {col}" for col in ['Title', 'Salary']])
elif header == 'Contact':
combined_headers.extend([f"{header} - {col}" for col in ['Office', 'Extn.', 'Email']])
# Extract data from table body
rows = []
for row in table.select('tbody tr'):
cells = [cell.text.strip() for cell in row.find_all('td')]
rows.append(cells)
df = pd.DataFrame(rows, columns=combined_headers)
return df
# Scrape the table
result_df = scrape_complex_table(url)
# Display the first few rows of the result
print(result_df.head())
result_df.to_csv('complex_table_data.csv', index=False)
print("Data has been saved to 'complex_table_data.csv'")Note: Make sure you have replaced ‘your_api_key_here’ with your actual API key before running the script.
Scraping Paginated HTML Tables with Python
When dealing with large datasets, tables are often divided across multiple pages to improve loading times and user experience. Traditionally, this would require setting up a headless browser with tools like Selenium. However, we can achieve the same results more efficiently using advanced rendering capabilities.
Note: Check this comprehensive tutorial on web scraping with Selenium to learn more.
Understanding Pagination Handling
From our established example, the table is paginated with “>” and “<” navigation buttons.
To scrape all the data, we need to:
- Load the initial page
- Click the “>” button
- Wait for new data to load
- Repeat until all pages are processed
Using Advanced Render Instructions
Instead of manually controlling a browser, we can send instructions to a headless browser through an API.
The Render Instruction Set allows you to send instructions to a headless browser via an API call, guiding it on what actions to perform during page rendering. These instructions are sent as a JSON object in the API request headers.
Let’s demonstrate how to scrape a paginated table using these render instructions:
Configuring API and Render Instructions
First, we’ll set up our API key and the target URL we want to scrape. Remember to replace ‘your_api_key’ with your actual API key.
api_key = 'your_api_key' # Replace with your actual API key
target_url = 'https://datatables.net/examples/styling/stripe.html'Now, we’ll define the set of render instructions.
# Configuration for render instructions
config = [{
"type": "loop",
"for": 5, # Number of times to execute the instructions
"instructions": [
{
"type": "click",
"selector": {
"type": "css",
"value": "button.dt-paging-button.next"
}
},
{
"type": "wait",
"value": 3 # Wait time in seconds after clicking
}
]
}]The loop instruction repeats the set of instructions a specified number of times (“for”: 5). The click instruction simulates a click on the “>” button to navigate to the next page.
Making the Request
After defining the render instructions, we need to convert the config dictionary to a JSON string because the API requires the instructions to be in JSON format when included in the request headers.
config_json = json.dumps(config)We then prepare the headers and payload for the GET request:
# Headers to include API instructions
headers = {
'x-sapi-api_key': api_key,
'x-sapi-render': 'true',
'x-sapi-instruction_set': config_json
}
# Payload with the target URL
payload = {'url': target_url}In the headers:
'x-sapi-api_key'is where you include your API key for authentication.'x-sapi-render'is set to ‘true’ to enable rendering with a headless browser, allowing the execution of JavaScript and dynamic content loading.'x-sapi-instruction_set'contains the render instructions in JSON format, which we previously converted withjson.dumps(config).
The payload simply includes the ‘url’ key with the target_url value, indicating the webpage we want to scrape.
Processing the Table Data
Once we have the response, we can process the table data using BeautifulSoup:
# Parse the HTML content
soup = BeautifulSoup(response.text, 'html.parser')
employee_list = []
# Find and process the table
table = soup.find('table', class_='stripe')
# Extract data from all rows
for employee_data in table.find_all('tbody'):
rows = employee_data.find_all('tr')
for row in rows:
cells = row.find_all('td')
employee_list.append({
'Name': cells[0].text,
'Position': cells[1].text,
'Office': cells[2].text,
'Age': cells[3].text,
'Start date': cells[4].text,
'salary': cells[5].text
})
# Save the data to a JSON file
with open('employee_data.json', 'w') as json_file:
json.dump(employee_list, json_file, indent=2)Benefits of Using Render Instructions
Using render instructions offers several advantages over traditional browser automation:
- No need to install and manage Selenium or a WebDriver
- Simpler code with fewer dependencies
- Better handling of anti-bot measures through the API
- More reliable execution with built-in waits and retries
- Easy deployment to servers without browser dependencies
Dealing with Errors while Scraping HTML Tables
HTML tables on real websites often have complex layouts, making them challenging for beginners to scrape. These tables may include mixed data types, nested elements, merged cells, and other intricate structures that complicate table parsing during scraping.
Let’s explore some common issues and their solutions to make your table scraping more efficient and reliable:
1. Handling Empty Cells and Missing Data
Empty cells or missing data can cause your scraping script to fail or produce incomplete results. Here’s how to handle them gracefully:
def extract_cell_data(cell):
# Handle empty cells
if not cell:
return "N/A"
# Handle cells with only whitespace
if cell.text.strip() == "":
return "N/A"
return cell.text.strip()2. JavaScript-Injected Tables
Some tables are dynamically generated using JavaScript, meaning the data isn’t present in the initial HTML response but is injected into the page after being rendered by a browser. Traditional scraping methods may fail to retrieve this content since they don’t execute JavaScript.
Advanced render instructions allow you to simulate user interactions and execute JavaScript within a headless browser environment. This enables you to scrape dynamically loaded tables without resorting to complex tools like Selenium.
[
{
"type": "input",
"selector": { "type": "css", "value": "#searchInput" },
"value": "cowboy boots"
},
{
"type": "click",
"selector": {
"type": "css",
"value": "#search-form button[type=\"submit\"]"
}
},
{
"type": "wait_for_selector",
"selector": { "type": "css", "value": "#content" }
}
]Note: To learn more about scraping javascript tables, kindly refer to our comprehensive guide.
3. Malformed HTML Tables
Some tables might have invalid HTML structure or missing closing tags. A better parser to use in this instance would be html5lib. Here’s how to handle them:
def clean_table_html(html_content):
# Use html5lib parser for better handling of malformed HTML
soup = BeautifulSoup(html_content, 'html5lib')
# Function to check if table is valid
def is_valid_table(table):
if not table.find('tr'):
return False
rows = table.find_all('tr')
if not rows:
return False
return True
# Find all tables and process only valid ones
tables = []
for table in soup.find_all('table'):
if is_valid_table(table):
# Clean up any invalid nested tables
for nested_table in table.find_all('table'):
nested_table.decompose()
tables.append(table)
return tables4. Using Pandas over Other Libraries
Using Pandas for scraping HTML tables saves considerable time and makes code more reliable because you select the entire table, not individual items that may change over time.
The read_html method lets you directly fetch tables without parsing the entire HTML document. It’s significantly faster for extracting tables since it’s optimized for this specific task. It also directly returns a DataFrame, which makes it easy to clean, transform, and analyze the data.
Scraping HTML Tables Using Pandas
Before you finish reading, let’s explore an alternative approach to scrape HTML tables. In just a few lines of code, we can scrape all tabular data from an HTML document and store it in a dataframe using Pandas.
Create a new folder inside the project’s directory (we named it pandas-html-table-scraper) and create a new file called pandas_table_scraper.py.
Let’s open a new terminal and navigate to the folder we just created (cd pandas-html-table-scraper) and install pandas:
pip install pandasImport it at the top of the file.
import pandas as pdPandas has a function called read_html() which essentially scrapes the target URL for us and returns all HTML tables as a list of DataFrame objects.
However, for this to work effectively, the HTML table needs to be structured reasonably well, as the function will look for elements like <table> to identify the tables in the file.
To use the function, let’s create a new variable and pass the URL we used previously:
employee_data = pd.read_html('http://api.webparsers.com?api_key=51e43be283e4db2a5afbxxxxxxxxxxxx&url=https://datatables.net/examples/styling/stripe.html')When printed, it’ll return a list of HTML tables within the page.
If we compare the first three rows in the DataFrame, they’re a perfect match to what we scraped with Beautiful Soup.
To work with JSON, Pandas has a built-in .to_json() function. It’ll convert a list of DataFrame objects into a JSON string.
All we need to do is call the method on our DataFrame and pass in the path, the format (split, data, records, index, etc.) and add the indent to make it more readable:
employee_data[0].to_json('./employee_list.json', orient='index', indent=2)If we run our code now, here’s the resulting file output.
Notice that we needed to select our table from the index ([0]) because .read_html() returns a list, not a single object.
Here’s the complete code for your reference:
import pandas as pd
employee_data = pd.read_html('http://api.webparsers.com?api_key=51e43be283e4db2a5afbxxxxxxxxxxxx&url=https://datatables.net/examples/styling/stripe.html')
employee_data[0].to_json('./employee_list.json', orient='index', indent=2)Armed with this knowledge, you’re ready to start scraping virtually any HTML table on the web. Just remember that if you understand how the website is structured and the logic behind it, there’s nothing you can’t scrape.
That said, these methods will only work as long as the data exists within the HTML file. If you encounter a dynamically generated table, you’ll need a different approach. For these types of tables, we’ve created a comprehensive guide to scraping JavaScript tables with Python without requiring headless browsers.
Until next time, happy scraping!