In this comprehensive web scraping tutorial, we’ll explore how to extract job listing data from Indeed.com, one of the most popular job search platforms available today. The process is surprisingly straightforward and efficient!
We’ll construct our scraper using just a few lines of Python code, examining how Indeed’s search functionality operates to replicate it in our scraper and extract job data from embedded JavaScript variables. Let’s get started!
Key Takeaways
Master indeed scraper development using Python with httpx and parsel, extracting job listings from embedded JavaScript variables and handling Indeed’s search parameters for comprehensive job data collection.
- Reverse engineer Indeed’s search API endpoints by intercepting browser network requests and analyzing JSON response structures
- Parse embedded JavaScript variables containing job data using regex patterns and JSON extraction techniques
- Implement search parameter management and pagination handling for comprehensive job data collection across multiple pages
- Extract structured job information including titles, companies, locations, and salary data from JSON responses
- Handle Indeed’s anti-scraping measures with realistic headers and request spacing to avoid detection
- Configure concurrent request processing and response parsing for efficient large-scale job data extraction
Why Scrape Indeed.com?
The employment landscape is constantly evolving, with fresh opportunities and changes occurring daily. By scraping Indeed, you can receive real-time updates for various job postings across different industries and geographical locations.
Job data scraping from Indeed also enables comprehensive market analysis. By aggregating employment data, you can identify emerging patterns, high-demand skills, and evolving job requirements across various sectors.
Furthermore, manually browsing through thousands of job listings on the platform can be extremely time-consuming. Through automated scraping, we can efficiently collect Indeed listings or establish customized job posting notification systems.
Project Setup
For this web scraper implementation, we’ll only require an HTTP client library such as httpx, which can be installed through the pip console command:
$ pip install httpxWhile there are numerous HTTP clients available in Python like requests, httpx, aiohttp, etc., we recommend httpx as it’s the least likely to be blocked, since it supports the http2 protocol. Additionally, httpx enables us to execute our web scraping code asynchronously, significantly improving our scraping performance.
For Webparsers users, we’ll also be providing code versions using scrapfly-sdk.
Web Scraping with Python
Introduction tutorial to web scraping with Python. How to collect and parse public data. Challenges, best practices and an example project.
Finding Indeed Jobs
To begin, let’s examine how we can locate job listings on Indeed.com.
Navigate to the website homepage, submit a search query, and you will be redirected to a search URL containing several key parameters:
https://www.indeed.com/jobs?q=python&l=TexasFor instance, to discover Python jobs in Texas, we simply need to send a request with l=Texas and q=Python URL parameters:
Python
import httpx
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36",
"Accept-Encoding": "gzip, deflate, br",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Connection": "keep-alive",
"Accept-Language": "en-US,en;q=0.9,lt;q=0.8,et;q=0.7,de;q=0.6",
}
response = httpx.get("https://www.indeed.com/jobs?q=python&l=Texas", headers=HEADERS)
print(response)Note: if you receive response status code 403 here, it’s likely you are being blocked. Run the Webparsers code tabs to avoid blocking.
We obtained a single page containing 15 job listings! Before collecting the remaining pages, let’s explore how we can parse job listing data from this response.
While we could parse the HTML document using CSS or XPath selectors, there’s a more efficient approach: we can locate all the job listing data embedded deep within the HTML as a JSON document:
page source of indeed.com search page embedded data
This type of information is commonly referred to as hidden web data. It represents the same data displayed on the web page but in its pre-rendered HTML state.
Let’s parse this data using a straightforward regular expression pattern:
Python
import httpx
import re
import json
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36",
"Accept-Encoding": "gzip, deflate, br",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Connection": "keep-alive",
"Accept-Language": "en-US,en;q=0.9,lt;q=0.8,et;q=0.7,de;q=0.6",
}
def parse_search_page(html: str):
data = re.findall(r'window.mosaic.providerData\["mosaic-provider-jobcards"\]=(\{.+?\});', html)
data = json.loads(data[0])
return {
"results": data["metaData"]["mosaicProviderJobCardsModel"]["results"],
"meta": data["metaData"]["mosaicProviderJobCardsModel"]["tierSummaries"],
}
response = httpx.get("https://www.indeed.com/jobs?q=python&l=Texas", headers=HEADERS)
print(parse_search_page(response.text))In the code above, we utilize a regular expression pattern to select the mosaic-provider-jobcards variable value, load it as a Python dictionary, and extract the result and paging metadata.
Now that we have the initial page results and total page count, we can retrieve the remaining pages:
Python
import asyncio
import httpx
import json
import re
from urllib.parse import urlencode
def parse_search_page(html: str):
data = re.findall(r'window.mosaic.providerData\["mosaic-provider-jobcards"\]=(\{.+?\});', html)
data = json.loads(data[0])
return {
"results": data["metaData"]["mosaicProviderJobCardsModel"]["results"],
"meta": data["metaData"]["mosaicProviderJobCardsModel"]["tierSummaries"],
}
async def scrape_search(client: httpx.AsyncClient, query: str, location: str, max_results: int = 50):
def make_page_url(offset):
parameters = {"q": query, "l": location, "filter": 0, "start": offset}
return "https://www.indeed.com/jobs?" + urlencode(parameters)
print(f"scraping first page of search: {query=}, {location=}")
response_first_page = await client.get(make_page_url(0))
data_first_page = parse_search_page(response_first_page.text)
results = data_first_page["results"]
total_results = sum(category["jobCount"] for category in data_first_page["meta"])
# there's a page limit on indeed.com of 1000 results per search
if total_results > max_results:
total_results = max_results
print(f"scraping remaining {total_results - 10 / 10} pages")
other_pages = [make_page_url(offset) for offset in range(10, total_results + 10, 10)]
for response in await asyncio.gather(*[client.get(url=url) for url in other_pages]):
results.extend(parse_search_page(response.text))
return resultsWe’ve successfully scraped extensive amounts of data with remarkably few lines of Python code! Let’s learn how to scrape individual job pages to obtain the remaining details of a job listing, such as the complete description.
Scraping Indeed Jobs
Our search results contain nearly all job listing data except certain details, such as a complete job description. To scrape this information, we need the job ID, which is located in the jobkey field within our search results:
{
"jobkey": "a82cf0bd2092efa3",
}Using the jobkey, we can request the complete job details page, and similar to the search process, we can parse the hidden data instead of the HTML:
page source of indeed.com search page embedded data
We can observe that all job and page information is embedded in the _initialData variable. It can be extracted using a simple regular expression pattern:
Python
import re
import json
import httpx
import asyncio
from typing import List
def parse_job_page(html):
"""parse job data from job listing page"""
data = re.findall(r"_initialData=(\{.+?\});", html)
data = json.loads(data[0])
return data["jobInfoWrapperModel"]["jobInfoModel"]
async def scrape_jobs(client: httpx.AsyncClient, job_keys: List[str]):
"""scrape job details from job page for given job keys"""
urls = [f"https://www.indeed.com/m/basecamp/viewjob?viewtype=embedded&jk={job_key}" for job_key in job_keys]
scraped = []
for response in await asyncio.gather(*[client.get(url=url) for url in urls]):
scraped.append(parse_job_page(response.text))
return scrapedWhen we run this scraper, we should see the complete job description printed out.
With this final feature, our Indeed scraper is ready for deployment! However, our scraper is very likely to encounter blocking issues when running at scale. For that reason, let’s examine how we can integrate Webparsers to avoid being blocked.
Bypass Indeed Blocking with Webparsers
Indeed.com employs anti-scraping protection to block web scraper traffic. To circumvent this, we can use Webparsers web scraping API which will help you scale up effectively!
scrapfly middleware
Webparsers provides web scraping, screenshot, and extraction APIs for data collection at scale.
- Anti-bot protection bypass – scrape web pages without blocking!
- Rotating residential proxies – prevent IP address and geographic blocks.
- JavaScript rendering – scrape dynamic web pages through cloud browsers.
- Full browser automation – control browsers to scroll, input and click on objects.
- Format conversion – scrape as HTML, JSON, Text, or Markdown.
- Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.
For our Indeed scraper, we’ll be using the Anti Scraping Protection Bypass feature via scrapfly-sdk, which can be installed using the pip console command:
$ pip install scrapfly-sdkNow, we can enable the Anti Scraping Protection bypass via the asp=True flag:
from scrapfly import ScrapflyClient, ScrapeConfig
client = ScrapflyClient(key="YOUR_API_KEY")
result = client.scrape(ScrapeConfig(
url="https://www.indeed.com/jobs?q=python&l=Texas",
asp=True,
# ^ enable Anti Scraping Protection
))
html = result.content # get the page HTML
selector = result.selector # use the built-in parsel selectorFAQ
Is it legal to scrape Indeed.com?
Yes. The job data on Indeed.com is publicly available so it’s perfectly legal to scrape. Note that some of the scraped material can be protected by copyright, such as images.
Can Indeed be scraped using headless browsers such as Playwright?
Yes, but as covered in this article it’s not necessary. Indeed pages are powered by a JSON API which can be scraped directly. This reduces the resource requirement for both the scraper and Indeed.com public data servers.
Is there a public API for Indeed.com?
No. As of the time of writing, there is no public API for Indeed.com job data. However, as demonstrated in this article, Indeed.com can be easily scraped using Python!
Indeed Scraping Summary
In this comprehensive web scraping tutorial, we’ve explored web scraping Indeed.com job listing search functionality.
We constructed a search URL using custom search parameters and parsed job data from embedded JSON data using regular expressions. As an additional feature, we also examined scraping complete job listing descriptions and how to avoid blocking using the scrapfly SDK.
Legal Disclaimer and Precautions
This tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect and here’s a good summary of what not to do:
- Do not scrape at rates that could damage the website.
- Do not scrape data that’s not available publicly.
- Do not store PII of EU citizens who are protected by GDPR.
- Do not repurpose the entire public datasets which can be illegal in some countries.
Webparsers does not offer legal advice but these are good general rules to follow in web scraping and for more you should consult a lawyer.