Yelp.com stands as one of the most popular platforms for business directories. It contains comprehensive company details, including addresses, websites, and locations, along with user reviews.
In this web scraping tutorial, we’ll demonstrate how to scrape yelp.com using Python. We’ll begin by reverse engineering the search functionality to discover businesses. Then, we’ll scrape and parse business data and reviews. Finally, we’ll explore methods to avoid Yelp web scraping blocking when operating at scale.
Key Takeaways
- Learn to scrape Yelp.com business data and reviews using Python with httpx and parsel, handling search functionality and anti-bot measures for comprehensive business directory extraction.
- Use Yelp’s search API endpoints to access business listings and review data without JavaScript rendering
- Parse JSON responses with jmespath to extract structured business and review information efficiently
- Handle Yelp’s anti-scraping measures with realistic headers, user agents, and request spacing
- Extract comprehensive business data including ratings, reviews, addresses, and contact information
- Implement proper error handling and retry logic for rate limiting and temporary blocking scenarios
- Use Webparsers SDK for automated Yelp scraping with anti-blocking and geographic targeting features
Why Scrape Yelp?
Yelp ranks among the largest websites for business directories. It encompasses various businesses across different categories, from restaurants to local service providers. Consequently, scraping Yelp data proves valuable for researchers seeking to understand market trends and analyze competitors.
Yelp also hosts thousands of detailed reviews. Business owners can scrape Yelp reviews and leverage machine learning techniques to analyze user opinions and evaluate customer experience.
Additionally, if you’re an individual explorer on Yelp, navigating through countless reviews can be tedious and time-consuming. Scraping Yelp allows you to retrieve thousands of data points quickly with more precise search capabilities.
Project Setup
To scrape Yelp, we’ll utilize several Python community packages:
- httpx – An HTTP client we’ll use to request Yelp pages.
- parsel – HTML parsing library we’ll use for parsing HTML using selectors like XPath and CSS.
- asyncio – For running our Yelp scraper code asynchronously, increasing our web scraping speed.
- JMESPath – For parsing and refining JSON datasets to exclude unnecessary data.
Since asyncio comes pre-installed in Python, we’ll only need to install the other libraries using the following pip command:
$ pip install httpx parsel jmespath
Alternatively, feel free to swap httpx with any other HTTP client package such as requests, as we’ll only need basic HTTP functions available across different HTTP clients. As for parsel, another excellent alternative is the beautifulsoup package.
Discovering Yelp Company Pages
Before scraping Yelp, let’s identify a method to discover businesses on the website. Examining Yelp’s robots.txt instructions reveals that it doesn’t provide sitemaps or directory pages. Therefore, to navigate the website, we must reverse engineer their search functionality and replicate it within our scraper.
How to Scrape Yelp Search
Let’s start by submitting a search query and examining the results:
Upon entering search details, we get redirected to a URL with the following search parameters:
https://www.yelp.com/search?find_desc=plumbers&find_loc=Toronto%2C+Ontario%2C+Canada&ns=1&start=220
We’ll use these URL parameters to navigate search pages. But first, let’s parse the search result data by extracting the webpage’s hidden data.
To locate the hidden web data on search pages, follow these steps:
- Open the browser developer tools by pressing the F12 key
- Search for the script tag with
data-id='react-root-props'
Following these steps reveals the target script tag containing the search data in JSON format.
To scrape search pages, we’ll select the script tag containing the data and parse its JSON content:
import asyncio
import json
import httpx
async def _search_yelp_page(keyword: str, location: str, session: httpx.AsyncClient, offset=0):
"""scrape single page of yelp search"""
# final url example:
# https://www.yelp.com/search/snippet?find_desc=plumbers&find_loc=Toronto%2C+Ontario%2C+Canada&ns=1&start=210&parent_request_id=54233ce74d09d270&request_origin=user
resp = await session.get(
"https://www.yelp.com/search/snippet",
params={
"find_desc": keyword,
"find_loc": location,
"start": offset,
"parent_request": "",
"ns": 1,
"request_origin": "user"
}
)
assert resp.status_code == 200, "request is blocked, refer to bypassing Yelp scraping blocking section"
return json.loads(resp.text)
Note: Yelp is known for its high blocking rate, and you’re likely to get blocked while running the code. To avoid blocking, consider using the Webparsers code version for reliable scraping.
In the above code, we replicate the request used for searching. Next, let’s define the logic responsible for selecting the script tag containing the hidden data and parsing it:
def parse_search(response: Response):
"""parse listing data from the search XHR data"""
search_data = []
selector = Selector(text=response.text)
script = selector.xpath("//script[@data-id='react-root-props']/text()").get()
data = json.loads(script.split("react_root_props = ")[-1].rsplit(";", 1)[0])
for item in data["legacyProps"]["searchAppProps"]["searchPageProps"]["mainContentComponentsListProps"]:
# filter search data cards
if "bizId" in item.keys():
search_data.append(item)
# filter the max results count
elif "totalResults" in item["props"]:
total_results = item["props"]["totalResults"]
return {"search_data": search_data, "total_results": total_results}
Here, we filter out unnecessary metadata, such as ads and tracking data, to retain only the actual search data. Finally, let’s complete this part of our Yelp scraper with iteration logic to scrape all available search pages. We’ll begin by scraping the first search page and then scrape the remaining pages concurrently:
import json
import math
import asyncio
from parsel import Selector
from typing import List, Dict
from urllib.parse import urlencode
from httpx import AsyncClient, Response
# initialize an async httpx client
client = AsyncClient(
# enable http2
http2=True,
# add basic browser like headers to prevent getting blocked
headers={
"Accept-Language": "en-US,en;q=0.9",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, br",
"Cookie": "intl_splash=false"
},
follow_redirects=True
)
def parse_search(response: Response) -> List[Dict]:
"""parse listing data from the search XHR data"""
search_data = []
selector = Selector(text=response.text)
script = selector.xpath("//script[@data-id='react-root-props']/text()").get()
data = json.loads(script.split("react_root_props = ")[-1].rsplit(";", 1)[0])
for item in data["legacyProps"]["searchAppProps"]["searchPageProps"]["mainContentComponentsListProps"]:
# filter search data cards
if "bizId" in item.keys():
search_data.append(item)
# filter the max results count
elif "totalResults" in item["props"]:
total_results = item["props"]["totalResults"]
return {"search_data": search_data, "total_results": total_results}
async def scrape_search(keyword: str, location: str, max_pages: int = None):
"""scrape single page of yelp search"""
def make_search_url(offset):
base_url = "https://www.yelp.com/search?"
params = {"find_desc": keyword, "find_loc": location, "start": offset}
return base_url + urlencode(params)
# final url example:
# https://www.yelp.com/search?find_desc=plumbers&find_loc=Seattle%2C+WA&start=1
print("scraping the first search page")
first_page = await client.get(make_search_url(1))
data = parse_search(first_page)
search_data = data["search_data"]
total_results = data["total_results"]
# find total page count to scrape
total_pages = math.ceil(total_results / 10) # each page contains 10 results
if max_pages and max_pages < total_pages:
total_pages = max_pages
# add the remaining pages to a scraping list and scrape them concurrently
print(f"scraping search pagination, remaining ({total_pages - 1}) more pages")
other_pages = [
client.get(make_search_url(offset))
for offset in range(11, total_pages * 10, 10)
]
for response in asyncio.as_completed(other_pages):
response = await response
assert response.status_code == 200, "request is blocked"
search_data.extend(parse_search(response)["search_data"])
print(f"scraped {len(search_data)} listings from search pages")
return search_data
Above, we employ a common pagination pattern used to accelerate web scraping through asynchronous requests. We retrieve the first page for the total page count, then schedule concurrent requests for the remaining pages.
We can successfully scrape Yelp for search data. In the following section, we’ll scrape company pages using each company URL.
How to Scrape Yelp Company Data
Let’s begin by examining the company pages to identify the data location within the HTML:
From the company page, we can see the HTML contains all the necessary data. However, the HTML structure presents challenges:
The class names are dynamically generated, meaning these class names are subject to change—making our Yelp scraper potentially unreliable.
Instead of relying on these class names, we’ll use more robust techniques, such as matching by text or using strict element values.
Fortunately, XPath allows for various selector tricks, such as the contains() and .. features:
//a[contains(text(),"Get Directions")]/../following-sibling::p/text()
We’ll use this parsing approach to extract the necessary company data. To evaluate the selectors with the HTML we receive, we’ll use parsel:
import httpx
import asyncio
import json
from typing import List, Dict
from parsel import Selector
def parse_company(resp: httpx.Response):
sel = Selector(text=resp.text)
xpath = lambda xp: sel.xpath(xp).get(default="").strip()
open_hours = {}
for day in sel.xpath('//th/p[contains(@class,"day-of-the-week")]'):
name = day.xpath('text()').get().strip()
value = day.xpath('../following-sibling::td//p/text()').get().strip()
open_hours[name.lower()] = value
return dict(
name=xpath('//h1/text()'),
website=xpath('//p[contains(text(),"Business website")]/following-sibling::p/a/text()'),
phone=xpath('//p[contains(text(),"Phone number")]/following-sibling::p/text()'),
address=xpath('//a[contains(text(),"Get Directions")]/../following-sibling::p/text()'),
logo=xpath('//img[contains(@class,"businessLogo")]/@src'),
claim_status="".join(sel.xpath('//span[span[contains(@class,"claim")]]/text()').getall()).strip().lower(),
open_hours=open_hours
)
async def scrape_yelp_companies(company_urls: List[str], session: httpx.AsyncClient) -> List[Dict]:
"""Scrape yelp company details from given yelp company urls"""
responses = await asyncio.gather(*[
session.get(url) for url in company_urls
])
results = []
for resp in responses:
results.append(parse_company(resp))
return results
Here, we define a parse_company function that uses XPath queries to capture the data fields we highlighted earlier. Then, we utilize this function with the scrape_yelp_companies function, which requests the Yelp company pages.
How to Scrape Yelp Reviews
To scrape Yelp company reviews, we’ll utilize another hidden API. To find this API, follow these steps:
- Go to any company page on Yelp.
- Open the browser developer tools by pressing the F12 key.
- Click the next review page.
After following these steps, you’ll find the reviews API recorded in the network tab.
The API call represents a GraphQL request. It uses several payload values, including the BUSINESS_ID to fetch review data.
To extract review data, we’ll replicate this GraphQL request within our Yelp scraper. First, we need to obtain the BUSINESS_ID from the company page.
How to Find Yelp’s Business ID
The Yelp company business ID can be found in the HTML source of the business page itself. Scraping it is straightforward:
import httpx
from parsel import Selector
def scrape_business_id(url):
response = httpx.get(url)
selector = Selector(response.text)
return selector.css('meta[name="yelp-biz-id"]::attr(content)').get()
print(scrape_business_id("https://www.yelp.com/biz/vons-1000-spirits-seattle-4"))
# Output: "Lw7NmZ3j-WEye97ywEmkXQ"
Next, we’ll define a request_reviews_api function to replicate the GraphQL request:
async def request_reviews_api(url: str, start_index: int, business_id):
"""request the graphql API for review data"""
pagination_data = {
"version": 1,
"type": "offset",
"offset": start_index
}
pagination_data = json.dumps(pagination_data)
after = base64.b64encode(pagination_data.encode('utf-8')).decode('utf-8')
payload = json.dumps([
{
"operationName": "GetBusinessReviewFeed",
"variables": {
"encBizId": business_id,
"reviewsPerPage": 10,
"selectedReviewEncId": "",
"hasSelectedReview": False,
"sortBy": "DATE_DESC",
"languageCode": "en",
"ratings": [5, 4, 3, 2, 1],
"isSearching": False,
"after": after,
"isTranslating": False,
"translateLanguageCode": "en",
"reactionsSourceFlow": "businessPageReviewSection",
"minConfidenceLevel": "HIGH_CONFIDENCE",
"highlightType": "",
"highlightIdentifier": "",
"isHighlighting": False
},
"extensions": {
"operationType": "query",
"documentId": "ef51f33d1b0eccc958dddbf6cde15739c48b34637a00ebe316441031d4bf7681"
}
}
])
headers = {
'authority': 'www.yelp.com',
'accept': '*/*',
'accept-language': 'en-US,en;q=0.9',
'cache-control': 'no-cache',
'content-type': 'application/json',
'origin': 'https://www.yelp.com',
'referer': url,
'x-apollo-operation-name': 'GetBusinessReviewFeed'
}
client = httpx.AsyncClient(timeout=10.0)
response = await client.post(
url="https://www.yelp.com/gql/batch",
json=payload,
headers=headers
)
return response
The above request uses basic headers and a JSON payload representing the GraphQL query. The payload includes pre-configured parameters required by the server along with two configurable parameters:
- encBizId: The business ID for the company page.
- after: An encoded JSON object that controls the reviews offset, which we’ll use to paginate through review pages.
Now that our function for requesting the reviews API is ready, let’s use it to crawl the reviews data:
import base64
import asyncio
import jmespath
import httpx
import json
from typing import List, Dict
from parsel import Selector
def parse_review_data(response: httpx.Response):
"""parse review data from the JSON response"""
data = json.loads(response.text)
reviews = data[0]["data"]["business"]["reviews"]["edges"]
parsed_reviews = []
for review in reviews:
result = jmespath.search(
"""{
encid: encid,
text: text.{full: full, language: language},
rating: rating,
feedback: feedback.{coolCount: coolCount, funnyCount: funnyCount, usefulCount: usefulCount},
author: author.{encid: encid, displayName: displayName, displayLocation: displayLocation, reviewCount: reviewCount, friendCount: friendCount, businessPhotoCount: businessPhotoCount},
business: business.{encid: encid, alias: alias, name: name},
createdAt: createdAt.utcDateTime,
businessPhotos: businessPhotos[].{encid: encid, photoUrl: photoUrl.url, caption: caption, helpfulCount: helpfulCount},
businessVideos: businessVideos,
availableReactions: availableReactionsContainer.availableReactions[].{displayText: displayText, reactionType: reactionType, count: count}
}""",
review["node"]
)
parsed_reviews.append(result)
total_reviews = data[0]["data"]["business"]["reviewCount"]
return {"reviews": parsed_reviews, "total_reviews": total_reviews}
async def scrape_reviews(session: httpx.AsyncClient, url: str, max_reviews: int = None) -> List[Dict]:
# first find business ID from business URL
print("scraping the business id from the business page")
response_business = await session.get(url)
assert response_business.status_code == 200, "request is blocked"
selector = Selector(text=response_business.text)
business_id = selector.css('meta[name="yelp-biz-id"]::attr(content)').get()
print("scraping the first review page")
first_page = await request_reviews_api(url=url, business_id=business_id, start_index=1)
review_data = parse_review_data(first_page)
reviews = review_data["reviews"]
total_reviews = review_data["total_reviews"]
# find total page count to scrape
if max_reviews and max_reviews < total_reviews:
total_reviews = max_reviews
# next, scrape the remaining review pages
print(f"scraping review pagination, remaining ({total_reviews // 10}) more pages")
for offset in range(11, total_reviews, 10):
response = await request_reviews_api(url=url, business_id=business_id, start_index=offset)
new_review_data = parse_review_data(response)["reviews"]
reviews.extend(new_review_data)
print(f"scraped {len(reviews)} reviews from review pages")
return reviews
In the above code, we define two functions:
- parse_review_data: Parses review data from the API response and extracts the total number of available reviews. It also refines the review object to exclude unnecessary data using JMESPath.
- scrape_reviews: The main Yelp reviews scraping logic. It starts by retrieving the BUSINESS_ID from the company page, then uses
request_reviews_apito get the first review page data including the total number of reviews available. Finally, it crawls through the remaining review pages to extract the desired amount of reviews.
Although we specified the total number of reviews in our scraper to be only 30, we can scrape additional reviews in seconds. This is because requesting the API is much faster than requesting HTML pages!
Bypass Yelp Blocking with Webparsers
Web scraping Yelp is a very popular use case, and the website employs various techniques to block web scrapers.
To resist blocking, we replicated common browser headers with our scraper. However, we’re likely to get blocked as soon as we scale our scraping requests.
Once Yelp identifies the client as a web scraper, it redirects all requests to a blocked page. To avoid blocking in this project, we’ll use Webparsers’ web scraping API.
Webparsers provides web scraping, screenshot, and extraction APIs for data collection at scale:
- Anti-bot protection bypass – scrape web pages without blocking!
- Rotating residential proxies – prevent IP address and geographic blocks.
- JavaScript rendering – scrape dynamic web pages through cloud browsers.
- Full browser automation – control browsers to scroll, input and click on objects.
- Format conversion – scrape as HTML, JSON, Text, or Markdown.
- Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.
To scrape Yelp.com using Webparsers and Python, install the Webparsers Python SDK:
$ pip install webparsers-sdk
Then, replace the httpx client with Webparsers’ SDK. For example, here’s how to scrape business phone numbers on Yelp’s company page:
import httpx
from parsel import Selector
response = httpx.get("https://www.yelp.com/biz/smooth-air-brampton")
selector = Selector(text=response.text)
phone_number = selector.xpath('//p[contains(text(),"Phone number")]/following-sibling::p/text()').get()
# Using Webparsers SDK:
from webparsers import WebparsersClient, ScrapeConfig
client = WebparsersClient("YOUR WEBPARSERS KEY")
result = client.scrape(ScrapeConfig(
"https://www.yelp.com/biz/smooth-air-brampton",
# select specific proxy country
country="US",
# enable anti scraping protection bypass:
asp=True,
render_js=True # enable JS rendering if needed
))
phone_number = result.selector.xpath('//p[contains(text(),"Phone number")]/following-sibling::p/text()').get()
print(phone_number)
FAQ
Let’s address some frequently asked questions about web scraping Yelp:
Is web scraping Yelp legal?
Yes, all the data on Yelp.com is publicly available, and it’s legal to scrape as long as the scraping rate is reasonable and doesn’t cause harm to the website.
Is there a public API for Yelp?
At the time of writing, Yelp doesn’t offer APIs for public use. However, we did discover private APIs for company search and reviews, which we can utilize for web scraping.
Are there alternatives for Yelp?
Yes, several popular alternatives exist for business and review data, including Yellowpages, TripAdvisor for travel reviews, Trustpilot for product reviews, and Google Maps for local business data.
How to scrape Yelp reviews?
To retrieve specific company reviews on Yelp, you need to replicate a request to the reviews API. Open browser developer tools and click on the 2nd review page to inspect the outgoing requests. Refer to the scraping Yelp reviews section for detailed instructions.
Yelp Scraping Summary
In this guide, we explained how to scrape Yelp to retrieve company data. We also demonstrated how to utilize Yelp’s private APIs to scrape search and review data from the website.
For our Yelp scraper, we used Python with several community packages: httpx and parsel. To avoid Yelp scraping blocking, we used Webparsers—a smart API that configures every request’s connection to avoid blocking.
Legal Disclaimer and Precautions
This tutorial covers popular web scraping techniques for educational purposes. Interacting with public servers requires diligence and respect. Here’s a summary of best practices:
- Do not scrape at rates that could damage the website.
- Do not scrape data that’s not publicly available.
- Do not store PII of EU citizens protected by GDPR.
- Do not repurpose entire public datasets, which can be illegal in some countries.
Webparsers does not offer legal advice, but these are good general rules to follow in web scraping. For specific guidance, consult a lawyer.