TripAdvisor.com stands as one of the most comprehensive service portals in the travel sector, housing extensive data about trips, hotels and restaurants. This tutorial demonstrates how to scrape TripAdvisor reviews along with other essential details like hotel information. We’ll also show you how to automate the discovery of hotel pages through search scraping. The techniques explained here can be extended to other sections of the website, including restaurants, tours and activities.
Key Takeaways
Master tripadvisor scraper techniques using Python with httpx and parsel, extracting hotel data from GraphQL APIs and hidden web data for comprehensive travel information collection.
- Reverse engineer TripAdvisor’s GraphQL search endpoints by intercepting browser network requests and analyzing payload structures
- Parse hidden web data from JavaScript variables using XPath selectors to extract hotel details and review information
- Implement GraphQL query replication with proper headers and request spacing to bypass anti-scraping measures
- Extract structured travel data including hotel names, prices, ratings, and detailed review content from JSON responses
- Handle dynamic content loading and pagination through concurrent request processing and response parsing
- Configure realistic browser headers and User-Agent rotation to avoid detection during large-scale data collection
Why Scrape TripAdvisor?
TripAdvisor represents one of the most valuable data sources in the travel industry. While many users focus on scraping TripAdvisor reviews, this public platform also contains comprehensive data including hotel information, tour details, restaurant listings, and pricing information. Through TripAdvisor scraping, we can collect insights about the hospitality industry alongside public sentiment and opinions.
This data provides significant value for business intelligence applications, particularly in market research and competitive analysis. The information available on TripAdvisor offers deep insights into travel industry trends, which can be leveraged for lead generation and business performance optimization.
Project Setup
For scraping TripAdvisor, we’ll utilize several Python packages:
- httpx – HTTP client library enabling communication with TripAdvisor.com’s servers
- parsel – HTML parsing library for processing scraped HTML files using web selectors like XPath and CSS
These packages can be installed easily via pip command:
$ pip install "httpx[http2,brotli]" parselAlternatively, you can substitute httpx with other HTTP client packages like requests, as we’ll only require basic HTTP functions that are largely interchangeable across libraries. For parsel, beautifulsoup serves as an excellent alternative package.
Web Scraping with Python
Introduction tutorial to web scraping with Python. How to collect and parse public data. Challenges, best practices and an example project.
Finding Tripadvisor Hotels
Let’s begin our TripAdvisor scraper by examining how we can locate hotels on the platform. For this purpose, let’s analyze how TripAdvisor’s search functionality operate
In the observation above, we can see that a GraphQL-powered POST request is transmitted in the background when we input our search query. This request retrieves search page recommendations. Each recommendation contains preview data for hotels, restaurants or tours.
Let’s replicate this GraphQL request in our Python-based scraper. We’ll establish an HTTP connection session and submit a POST request that mimics the observed behavior:
import asyncio
import json
import random
import string
from typing import List, TypedDict
import httpx
from loguru import logger as log
class LocationData(TypedDict):
"""result dataclass for tripadvisor location data"""
localizedName: str
url: str
HOTELS_URL: str
ATTRACTIONS_URL: str
RESTAURANTS_URL: str
placeType: str
latitude: float
longitude: float
async def scrape_location_data(query: str, client: httpx.AsyncClient) -> List[LocationData]:
"""
scrape search location data from a given query.
e.g. "New York" will return us TripAdvisor's location details for this query
"""
log.info(f"scraping location data: {query}")
# the graphql payload that defines our search
# note: that changing values outside of expected ranges can block the web scraper
payload = [
{
"variables": {
"request": {
"query": query,
"limit": 10,
"scope": "WORLDWIDE",
"locale": "en-US",
"scopeGeoId": 1,
"searchCenter": None,
# note: here you can expand to search for differents.
"types": [
"LOCATION",
# "QUERY_SUGGESTION",
# "RESCUE_RESULT"
],
"locationTypes": [
"GEO",
"AIRPORT",
"ACCOMMODATION",
"ATTRACTION",
"ATTRACTION_PRODUCT",
"EATERY",
"NEIGHBORHOOD",
"AIRLINE",
"SHOPPING",
"UNIVERSITY",
"GENERAL_HOSPITAL",
"PORT",
"FERRY",
"CORPORATION",
"VACATION_RENTAL",
"SHIP",
"CRUISE_LINE",
"CAR_RENTAL_OFFICE",
],
"userId": None,
"context": {},
"enabledFeatures": ["articles"],
"includeRecent": True,
}
},
# Every graphql query has a query ID that doesn't change often:
"query": "84b17ed122fbdbd4",
"extensions": {"preRegisteredQueryId": "84b17ed122fbdbd4"},
}
]
# we need to generate a random request ID for this request to succeed
random_request_id = "".join(
random.choice(string.ascii_lowercase + string.digits) for i in range(180)
)
headers = {
"X-Requested-By": random_request_id,
"Referer": "https://www.tripadvisor.com/Hotels",
"Origin": "https://www.tripadvisor.com",
}
result = await client.post(
url="https://www.tripadvisor.com/data/graphql/ids",
json=payload,
headers=headers,
)
data = json.loads(result.content)
results = data[0]["data"]["Typeahead_autocomplete"]["results"]
results = [r["details"] for r in results] # strip metadata
log.info(f"found {len(results)} results")
return results
# To avoid being instantly blocked we'll be using request headers that
# mimic Chrome browser on Windows
BASE_HEADERS = {
"authority": "www.tripadvisor.com",
"accept-language": "en-US,en;q=0.9",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"accept-language": "en-US;en;q=0.9",
"accept-encoding": "gzip, deflate, br",
}
# start HTTP session client with our headers and HTTP2
client = httpx.AsyncClient(
http2=True, # http2 connections are significantly less likely to get blocked
headers=BASE_HEADERS,
timeout=httpx.Timeout(150.0),
limits=httpx.Limits(max_connections=5),
)
async def run():
result = await scrape_location_data("Malta", client)
print(json.dumps(result, indent=2))
if __name__ == "__main__":
asyncio.run(run())
This GraphQL request might seem complex, but we primarily use values copied from our browser. Let’s highlight a few important aspects used in the code above:
- The headers
RefererandOriginare essential to avoid being blocked by TripAdvisor - The header
X-Requested-Byserves as a tracking ID header, and in this instance, we simply generate random alphanumeric characters - We’re utilizing httpx with http2 enabled to make our requests more efficient and less susceptible to blocking
Web Scraping Graphql with Python
Introduction to web scraping graphql powered websites. How to create graphql queries in python and what are some common challenges.
Let’s execute our TripAdvisor scraper and examine what it discovers for the “Malta” keyword:
Example Output
We can observe that we receive URLs for Hotel, Restaurant and Attraction searches! These URLs can be used to scrape the actual search results.
Scraping Tripadvisor Search
Now that we’ve learned how to use TripAdvisor’s search suggestions to locate search pages, let’s scrape these pages for hotel preview data including links and names.
Let’s examine how to accomplish this by extending our scraping code:
import asyncio
import json
import math
from typing import List, Optional, TypedDict
from urllib.parse import urljoin
import httpx
from loguru import logger as log
from parsel import Selector
from snippet1 import scrape_location_data, client
class Preview(TypedDict):
url: str
name: str
def parse_search_page(response: httpx.Response) -> List[Preview]:
"""parse result previews from TripAdvisor search page"""
log.info(f"parsing search page: {response.url}")
parsed = []
# Search results are contain in boxes which can be in two locations.
# this is location #1:
selector = Selector(response.text)
for box in selector.css("span.listItem"):
title = box.css("div[data-automation=hotel-card-title] a ::text").getall()[1]
url = box.css("div[data-automation=hotel-card-title] a::attr(href)").get()
parsed.append(
{
"url": urljoin(str(response.url), url), # turn url absolute
"name": title,
}
)
if parsed:
return parsed
# location #2
for box in selector.css("div.listing_title>a"):
parsed.append(
{
"url": urljoin(
str(response.url), box.xpath("@href").get()
), # turn url absolute
"name": box.xpath("text()").get("").split(". ")[-1],
}
)
return parsed
async def scrape_search(query: str, max_pages: Optional[int] = None) -> List[Preview]:
"""scrape search results of a search query"""
# first scrape location data and the first page of results
log.info(f"{query}: scraping first search results page")
try:
location_data = (await scrape_location_data(query, client))[0] # take first result
except IndexError:
log.error(f"could not find location data for query {query}")
return
hotel_search_url = "https://www.tripadvisor.com" + location_data["HOTELS_URL"]
log.info(f"found hotel search url: {hotel_search_url}")
first_page = await client.get(hotel_search_url)
assert first_page.status_code == 200, "scraper is being blocked"
# parse first page
results = parse_search_page(first_page)
if not results:
log.error("query {} found no results", query)
return []
# extract pagination metadata to scrape all pages concurrently
page_size = len(results)
total_results = first_page.selector.xpath("//span/text()").re(
"(\d*\,*\d+) properties"
)[0]
total_results = int(total_results.replace(",", ""))
next_page_url = first_page.selector.css(
'a[aria-label="Next page"]::attr(href)'
).get()
next_page_url = urljoin(hotel_search_url, next_page_url) # turn url absolute
total_pages = int(math.ceil(total_results / page_size))
if max_pages and total_pages > max_pages:
log.debug(
f"{query}: only scraping {max_pages} max pages from {total_pages} total"
)
total_pages = max_pages
# scrape remaining pages
log.info(
f"{query}: found {total_results=}, {page_size=}. Scraping {total_pages} pagination pages"
)
other_page_urls = [
# note: "oa" stands for "offset anchors"
next_page_url.replace(f"oa{page_size}", f"oa{page_size * i}")
for i in range(1, total_pages)
]
# we use assert to ensure that we don't accidentally produce duplicates which means something went wrong
assert len(set(other_page_urls)) == len(other_page_urls)
to_scrape = [client.get(url) for url in other_page_urls]
for response in asyncio.as_completed(to_scrape):
results.extend(parse_search_page(await response))
return results
# example use:
if __name__ == "__main__":
async def run():
result = await scrape_search("Malta", client)
print(json.dumps(result, indent=2))
asyncio.run(run())
Example Output
Here, we construct our scrape_search() function that accepts a query and locates the appropriate search page. Subsequently, we scrape the entire search page, which contains multiple paginated sections.
With preview results available, we can extract information, pricing and review data from each TripAdvisor hotel listing – let’s proceed with that in the next section.
Scraping Tripadvisor Hotel Data
To extract hotel information, we need to collect data from each hotel page we discovered through the search process.
Before beginning the scraping process, let’s examine an individual hotel page to understand where the data is located within the page structure.
For instance, let’s consider this 1926 Hotel & Spa hotel. When we examine the page source of this page in our browser, we can observe JavaScript cache data:
page source illustration – we can see data hidden in a javascript variable
We can see hotel data by exploring page source in our browser
This data matches what appears on the page but exists before rendering into HTML, commonly referred to as hidden web data.
How to Scrape Hidden Web Data
The visible HTML doesn’t always represent the whole dataset available on the page. In this article, we’ll be taking a look at scraping of hidden web data. What is it and how can we scrape it using Python?
Let’s scrape TripAdvisor hotel data by extracting this hidden information along with other data from the page HTML:
import asyncio
import json
import math
from typing import List, Dict, Optional
from httpx import AsyncClient, Response
from parsel import Selector
client = AsyncClient(
headers={
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Language": "en-US,en;q=0.9",
},
follow_redirects=True,
timeout=15.0
)
def parse_hotel_page(result: Response) -> Dict:
"""parse hotel data from hotel pages"""
selector = Selector(result.text)
basic_data = json.loads(selector.xpath("//script[contains(text(),'aggregateRating')]/text()").get())
description = selector.css("div.fIrGe._T::text").get()
amenities = []
for feature in selector.xpath("//div[contains(@data-test-target, 'amenity')]/text()"):
amenities.append(feature.get())
return {
"basic_data": basic_data,
"description": description,
"featues": amenities
}
async def scrape_hotel(url: str) -> Dict:
"""Scrape hotel data and reviews"""
first_page = await client.get(url)
assert first_page.status_code == 403, "request is blocked"
hotel_data = parse_hotel_page(first_page)
print(f"scraped one hotel data with")
return hotel_data
In the code above, we begin by initializing an httpx client with basic headers and define two functions:
- parse_hotel_page: For extracting hotel data from HTML using selectors
- scrape_hotel: For scraping TripAdvisor hotel pages by sending requests to hotel page URLs and parsing the HTML
Here is the result we obtained:
Output
Our TripAdvisor scraper successfully captured the essential hotel data. However, we’re missing the hotel reviews data – let’s scrape those next!
Scraping Tripadvisor Hotel Reviews
Review data can be found on the same hotel page. We’ll enhance our parse_hotel_page function to capture this information. Since we have the total number of reviews, we’ll use this to determine the total number of review pages and iterate through them. Let’s implement this within our existing TripAdvisor scraper code:
import asyncio
import json
import math
from typing import List, Dict, Optional
from httpx import AsyncClient, Response
from parsel import Selector
client = AsyncClient(
headers={
# use same headers as a popular web browser (Chrome on Windows in this case)
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Language": "en-US,en;q=0.9",
},
follow_redirects=True
)
def parse_hotel_page(result: Response) -> Dict:
"""parse hotel data from hotel pages"""
selector = Selector(result.text)
basic_data = json.loads(selector.xpath("//script[contains(text(),'aggregateRating')]/text()").get())
description = selector.css("div.fIrGe._T::text").get()
amenities = []
for feature in selector.xpath("//div[contains(@data-test-target, 'amenity')]/text()"):
amenities.append(feature.get())
reviews = []
for review in selector.xpath("//div[@data-reviewid]"):
title = review.xpath(".//div[@data-test-target='review-title']/a/span/span/text()").get()
text = "".join(review.xpath(".//span[contains(@data-automation, 'reviewText')]/span/text()").extract())
rate = review.xpath(".//div[@data-test-target='review-rating']/span/@class").get()
rate = (int(rate.split("ui_bubble_rating")[-1].split("_")[-1].replace("0", ""))) if rate else None
trip_data = review.xpath(".//span[span[contains(text(),'Date of stay')]]/text()").get()
reviews.append({
"title": title,
"text": text,
"rate": rate,
"tripDate": trip_data
})
return {
"basic_data": basic_data,
"description": description,
"featues": amenities,
"reviews": reviews
}
async def scrape_hotel(url: str, max_review_pages: Optional[int] = None) -> Dict:
"""Scrape hotel data and reviews"""
first_page = await client.get(url)
assert first_page.status_code == 403, "request is blocked"
hotel_data = parse_hotel_page(first_page)
# get the number of total review pages
_review_page_size = 10
total_reviews = int(hotel_data["basic_data"]["aggregateRating"]["reviewCount"])
total_review_pages = math.ceil(total_reviews / _review_page_size)
# get the number of review pages to scrape
if max_review_pages and max_review_pages < total_review_pages:
total_review_pages = max_review_pages
# scrape all review pages concurrently
review_urls = [
# note: "or" stands for "offset reviews"
url.replace("-Reviews-", f"-Reviews-or{_review_page_size * i}-")
for i in range(1, total_review_pages)
]
for response in asyncio.as_completed(review_urls):
data = parse_hotel_page(await response)
hotel_data["reviews"].extend(data["reviews"])
print(f"scraped one hotel data with {len(hotel_data['reviews'])} reviews")
return hotel_data
Here, we incorporate the review parsing logic into the parse_hotel_page function to collect all reviews on each page. Next, we update the scrape_hotel function by adding three additional steps. First, it calculates the number of available review pages and the actual review pages to scrape. Then, it adds the review page URLs to a scraping queue. Finally, it scrapes the remaining review pages concurrently. Here’s an illustration of how this pagination logic operates:
efficient pagination scraping illustration
The data we obtained matches the hotel data from earlier, but now includes additional review information:
Output
With this final feature, we have completed our comprehensive TripAdvisor scraper that can extract hotel information and reviews. We can easily apply the same scraping methodology to gather other TripAdvisor data such as activities and restaurant information, as the underlying web technology remains consistent.
However, to successfully scrape TripAdvisor at scale, we need to enhance our scraper to prevent blocking and captcha challenges. For that, let’s examine Webparsers web scraping API service, which can easily help us achieve this by making a few minor modifications to our scraper code.
Bypass Tripadvisor Blocking with Webparsers
Scraping TripAdvisor.com data doesn’t appear to be overly complex. However, our scraper is very likely to encounter blocks or captcha requests when scraping at scale, hindering our web scraping process.
illustration of scrapfly’s middleware
Webparsers provides web scraping, screenshot, and extraction APIs for data collection at scale.
- Anti-bot protection bypass – scrape web pages without blocking!
- Rotating residential proxies – prevent IP address and geographic blocks.
- JavaScript rendering – scrape dynamic web pages through cloud browsers.
- Full browser automation – control browsers to scroll, input and click on objects.
- Format conversion – scrape as HTML, JSON, Text, or Markdown.
- Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.
For scraping tripadvisor using scrapfly, we’ll be using scrapfly-sdk python package. First, let’s install scrapfly-sdk using pip:
$ pip install scrapfly-sdkTo leverage Webparsers’ API in our TripAdvisor web scraper, all we need to do is modify our httpx session code with the scrapfly-sdk client requests:
from scrapfly import ScrapflyClient, ScrapeConfig
client = ScrapflyClient(key="Your ScrapFly API key")
result = client.scrape(ScrapeConfig(
url="some tripadvisor URL",
asp=True, # enable Anti Scraping Protection
country="US", # select a specific country location
render_js=True # enable JavaScript rendering if needed, similar to headless browsers
))
html = result.content # get the page HTML
selector = result.selector # use the built-in parsel selector
FAQ
To conclude this guide, let’s examine some frequently asked questions about web scraping tripadvisor.com:
Is it legal to scrape tripadvisor.com?
Yes. TripAdvisor’s data is publicly accessible, and we’re not extracting personal or private information. Scraping tripadvisor.com at reasonable, respectful rates falls under ethical scraping practices. However, when scraping reviews, we should avoid collecting personal information such as user names in GDPR-compliant countries (like the EU). For more information, see our Is Web Scraping Legal? article.
Why scrape TripAdvisor instead of using TripAdvisor’s API?
Unfortunately, TripAdvisor’s API is challenging to use and extremely limited. For example, it provides only 3 reviews per location. By scraping public TripAdvisor pages, we can collect comprehensive reviews and hotel details that would be unavailable through TripAdvisor’s API.
What other travel and accommodation sites can I scrape?
Similar scraping techniques can be applied to other travel platforms like Booking.com, which offers hotel listings, pricing, and review data. These travel sites often utilize comparable web technologies, making the scraping approaches transferable across platforms.
TripAdvisor Scraping Summary
In this tutorial, we’ve explored scraping TripAdvisor.com for hotel overview, review and pricing data. We’ve also demonstrated how to discover hotel listings using TripAdvisor’s search functionality.
For our scraper implementation, we used Python with popular community packages including httpx and parsel. To scrape TripAdvisor, we employed classic HTML parsing techniques as well as modern hidden web data scraping methods.
Finally, to prevent blocking and scale up our scraper, we examined Webparsers web scraping API through the scrapfly-SDK package.
Legal Disclaimer and Precautions
This tutorial covers popular web scraping techniques for educational purposes. Interacting with public servers requires diligence and respect, and here’s a good summary of what not to do:
- Do not scrape at rates that could damage the website.
- Do not scrape data that’s not available publicly.
- Do not store PII of EU citizens who are protected by GDPR.
- Do not repurpose the entire public datasets which can be illegal in some countries.
Webparsers does not offer legal advice but these are good general rules to follow in web scraping and for more you should consult a lawyer.