Booking.com stands as the largest travel reservation platform globally, containing publicly accessible data on thousands of hotels, resorts, and vacation rentals worldwide.
This comprehensive guide explores how to scrape Booking.com using Python programming language. We’ll begin with an overview of the website’s core functionality, then replicate its behavior in Python to extract hotel information and pricing data.
Finally, we’ll examine essential tips, tricks, and common challenges encountered when web scraping Booking.com. Let’s get started!
Key Takeaways
Master the art of scraping Booking.com hotel data and pricing using Python with httpx and parsel, effectively handling dynamic content, anti-bot measures, and geographic restrictions for comprehensive travel data extraction.
- Reverse engineer Booking.com’s search API endpoints by intercepting browser network requests
- Handle dynamic content loading with proper request headers and HTTP/2 support for anti-blocking
- Parse hotel data including names, prices, ratings, and availability from search results and detail pages
- Implement geographic targeting and language preferences to access region-specific hotel data
- Bypass Booking.com’s anti-scraping measures with realistic headers, rate limiting, and proxy rotation
- Extract structured pricing data including currency conversion and seasonal rate variations
Project Setup
This tutorial utilizes Python with Webparsers’ Python SDK.
Installation is straightforward using the pip command below:
$ pip install scrapfly-sdkFinding Booking Hotels
Our initial step involves understanding how to discover hotel pages for data extraction. Booking.com provides multiple approaches to accomplish this task.
Using Sitemaps
Booking.com offers easy access through its comprehensive sitemap system. Sitemaps are compressed XML files containing all website URLs categorized by subject matter.
To locate sitemaps, visit the /robots.txt page where you’ll find Sitemap links:
Sitemap: https://www.booking.com/sitembk-airport-index.xml
Sitemap: https://www.booking.com/sitembk-articles-index.xml
Sitemap: https://www.booking.com/sitembk-attractions-index.xml
Sitemap: https://www.booking.com/sitembk-beaches-index.xml
Sitemap: https://www.booking.com/sitembk-beach-holidays-index.xml
Sitemap: https://www.booking.com/sitembk-cars-index.xml
Sitemap: https://www.booking.com/sitembk-city-index.xml
Sitemap: https://www.booking.com/sitembk-country-index.xml
Sitemap: https://www.booking.com/sitembk-district-index.xml
Sitemap: https://www.booking.com/sitembk-hotel-index.xml
Sitemap: https://www.booking.com/sitembk-landmark-index.xml
Sitemap: https://www.booking.com/sitembk-region-index.xml
Sitemap: https://www.booking.com/sitembk-tourism-index.xml
Sitemap: https://www.booking.com/sitembk-themed-city-villas-index.xml
Sitemap: https://www.booking.com/sitembk-themed-country-golf-index.xml
Sitemap: https://www.booking.com/sitembk-themed-region-budget-index.xmlThese URLs are organized by cities, landmarks, and themes. For example, examining /sitembk-hotel-index.xml reveals it subdivides into additional sitemaps since individual sitemaps can only contain 50,000 results:
<sitemapindex xmlns="http://www.google.com/schemas/sitemap/0.9">
<sitemap>
<loc>https://www.booking.com/sitembk-hotel-zh-tw.0037.xml.gz</loc>
<lastmod>2022-05-17</lastmod>
</sitemap>
<sitemap>
<loc>https://www.booking.com/sitembk-hotel-zh-tw.0036.xml.gz</loc>
<lastmod>2022-05-17</lastmod>
</sitemap>
...This reveals 1710 sitemaps representing approximately 85 million links to various hotel pages. While not all are unique (some duplicates exist), this represents the most efficient method for discovering hotel listing pages on Booking.com.
Using sitemaps provides efficiency but lacks flexibility. When scraping specific data, we typically have particular areas or categories in mind.
For scraping hotels in specific areas or with certain features, we need to leverage Booking.com’s search system instead.
Scraping Booking.com Search
Alternatively, we can utilize the search system on Booking.com just like human users do.
Booking’s search may seem complex initially due to lengthy URLs, but deeper investigation reveals its simplicity, as most URL parameters are optional. For example, consider this “Hotels in London” query:
https://www.booking.com/searchresults.html?label=gen173nr-1DCAEoggI46AdIM1gEaN0BiAEBmAExuAEKyAEF2AED6AEB-AECiAIBqAIDuAK-u5eUBsACAdICJGRlN2VhYzYyLTJiYzItNDE0MS1iYmY4LWYwZjkxNTc0OGY4ONgCBOACAQ
&sid=51b2c8cd7b3c8377e83692903e6f19ca
&sb=1
&sb_lp=1
&src=index
&src_elem=sb
&error_url=https%3A%2F%2Fwww.booking.com%2Findex.html%3Flabel%3Dgen173nr-1DCAEoggI46AdIM1gEaN0BiAEBmAExuAEKyAEF2AED6AEB-AECiAIBqAIDuAK-u5eUBsACAdICJGRlN2VhYzYyLTJiYzItNDE0MS1iYmY4LWYwZjkxNTc0OGY4ONgCBOACAQ%26sid%3D51b2c8cd7b3c8377e83692903e6f19ca%26sb_price_type%3Dtotal%26%26
&ss=London%2C+Greater+London%2C+United+Kingdom
&is_ski_area=
&ssne=London
&ssne_untouched=London
&checkin_year=2022
&checkin_month=6
&checkin_monthday=9
&checkout_year=2022
&checkout_month=6
&checkout_monthday=11
&group_adults=2
&group_children=0
&no_rooms=1
&b_h4u_keep_filters=
&from_sf=1
&search_pageview_id=f25c2a9ee3630134
&ac_suggestion_list_length=5
&ac_suggestion_theme_list_length=0
&ac_position=0
&ac_langcode=en
&ac_click_type=b
&dest_id=-2601889
&dest_type=city
&iata=LON
&place_id_lat=51.507393
&place_id_lon=-0.127634
&search_pageview_id=f25c2a9ee3630134
&search_selected=true
&ss_raw=LondonDespite numerous parameters, we can simplify this to essential ones in our Python web scraper! Now that we understand the available query parameters on search pages, let’s proceed to data extraction!
Search pages utilize dynamic scroll actions to load additional results. While simulating scroll actions with headless browser automation is possible, it’s not recommended. Since search pages can contain thousands of results, scrolling through numerous pages in a single session becomes impractical.
Instead, we’ll scrape the hidden APIs responsible for fetching search results during scrolling. To capture this API, follow these steps:
- Open browser developer tools by pressing F12
- Select the network tab and filter by Fetch/XHR requests
- Scroll down the page to load more data
Following these steps reveals a request sent to booking.com/dml/graphql:
This request represents a GraphQL call. To scrape it, we must extract the required request body from the search HTML page:
import os
import re
import json
import asyncio
from urllib.parse import urlencode
from collections import defaultdict
from datetime import datetime, timedelta
from typing import Dict, List, Optional, TypedDict
from scrapfly import ScrapeApiResponse, ScrapeConfig, ScrapflyClient
BASE_CONFIG = {
"asp": True,
"country": "US",
}
SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"])
def retrieve_graphql_body(result: ScrapeApiResponse) -> List[Dict]:
"""parse the graphql search query from the HTML and return the full graphql body"""
selector = result.selector
script_data = selector.xpath("//script[@data-capla-store-data='apollo']/text()").get()
json_script_data = json.loads(script_data)
keys_list = list(json_script_data["ROOT_QUERY"]["searchQueries"].keys())
second_key = keys_list[1]
search_query_string = second_key[len("search("):-1]
input_json_object = json.loads(search_query_string)
return {
"operationName": "FullSearch",
"variables": {
"input": input_json_object["input"],
"carouselLowCodeExp": False
},
"extensions": {},
"query": "query FullSearch($input: SearchQueryInput!, $carouselLowCodeExp: Boolean!) {\n searchQueries {\n search(input: $input) {\n ...FullSearchFragment\n __typename\n }\n __typename\n }\n}\n\nfragment FullSearchFragment on SearchQueryOutput {\n banners {\n ...Banner\n __typename\n }\n breadcrumbs {\n ... on SearchResultsBreadcrumb {\n ...SearchResultsBreadcrumb\n __typename\n }\n ... on LandingPageBreadcrumb {\n ...LandingPageBreadcrumb\n __typename\n }\n __typename\n }\n carousels {\n ...Carousel\n __typename\n }\n destinationLocation {\n ...DestinationLocation\n __typename\n }\n entireHomesSearchEnabled\n dateFlexibilityOptions {\n enabled\n __typename\n }\n flexibleDatesConfig {\n broadDatesCalendar {\n checkinMonths\n los\n startWeekdays\n losType\n __typename\n }\n dateFlexUseCase\n dateRangeCalendar {\n flexWindow\n checkin\n checkout\n __typename\n }\n __typename\n }\n filters {\n ...FilterData\n __typename\n }\n filtersTrackOnView {\n type\n experimentHash\n value\n __typename\n }\n appliedFilterOptions {\n ...FilterOption\n __typename\n }\n recommendedFilterOptions {\n ...FilterOption\n __typename\n }\n pagination {\n nbResultsPerPage\n nbResultsTotal\n __typename\n }\n tripTypes {\n ...TripTypesData\n __typename\n }\n results {\n ...BasicPropertyData\n ...MatchingUnitConfigurations\n ...PropertyBlocks\n ...BookerExperienceData\n priceDisplayInfoIrene {\n ...PriceDisplayInfoIrene\n __typename\n }\n licenseDetails {\n nextToHotelName\n __typename\n }\n isTpiExclusiveProperty\n propertyCribsAvailabilityLabel\n mlBookingHomeTags\n trackOnView {\n experimentTag\n __typename\n }\n __typename\n }\n searchMeta {\n ...SearchMetadata\n __typename\n }\n sorters {\n option {\n ...SorterFields\n __typename\n }\n __typename\n }\n zeroResultsSection {\n ...ZeroResultsSection\n __typename\n }\n rocketmilesSearchUuid\n previousSearches {\n ...PreviousSearches\n __typename\n }\n frontierThemes {\n ...FrontierThemes\n __typename\n }\n merchComponents {\n ...MerchRegionIrene\n __typename\n }\n wishlistData {\n numProperties\n __typename\n }\n seoThemes {\n id\n caption\n __typename\n }\n __typename\n}\n\nfragment BasicPropertyData on SearchResultProperty {\n acceptsWalletCredit\n basicPropertyData {\n accommodationTypeId\n id\n isTestProperty\n location {\n address\n city\n countryCode\n __typename\n }\n pageName\n ufi\n photos {\n main {\n highResUrl {\n relativeUrl\n __typename\n }\n lowResUrl {\n relativeUrl\n __typename\n }\n highResJpegUrl {\n relativeUrl\n __typename\n }\n lowResJpegUrl {\n relativeUrl\n __typename\n }\n __typename\n }\n __typename\n }\n reviewScore: reviews {\n score: totalScore\n reviewCount: reviewsCount\n totalScoreTextTag {\n translation\n __typename\n }\n showScore\n secondaryScore\n secondaryTextTag {\n translation\n __typename\n }\n showSecondaryScore\n __typename\n }\n externalReviewScore: externalReviews {\n score: totalScore\n reviewCount: reviewsCount\n showScore\n totalScoreTextTag {\n translation\n __typename\n }\n __typename\n }\n starRating {\n value\n symbol\n caption {\n translation\n __typename\n }\n tocLink {\n translation\n __typename\n }\n showAdditionalInfoIcon\n __typename\n }\n isClosed\n paymentConfig {\n installments {\n minPriceFormatted\n maxAcceptCount\n __typename\n }\n __typename\n }\n __typename\n }\n badges {\n caption {\n translation\n __typename\n }\n closedFacilities {\n startDate\n endDate\n __typename\n }\n __typename\n }\n customBadges {\n showSkiToDoor\n showBhTravelCreditBadge\n showOnlineCheckinBadge\n __typename\n }\n description {\n text\n __typename\n }\n displayName {\n text\n translationTag {\n translation\n __typename\n }\n __typename\n }\n geniusInfo {\n benefitsCommunication {\n header {\n title\n __typename\n }\n items {\n title\n __typename\n }\n __typename\n }\n geniusBenefits\n geniusBenefitsData {\n hotelCardHasFreeBreakfast\n hotelCardHasFreeRoomUpgrade\n sortedBenefits\n __typename\n }\n showGeniusRateBadge\n __typename\n }\n location {\n displayLocation\n mainDistance\n publicTransportDistanceDescription\n skiLiftDistance\n beachDistance\n nearbyBeachNames\n beachWalkingTime\n geoDistanceMeters\n __typename\n }\n mealPlanIncluded {\n mealPlanType\n text\n __typename\n }\n persuasion {\n autoextended\n geniusRateAvailable\n highlighted\n preferred\n preferredPlus\n showNativeAdLabel\n nativeAdId\n nativeAdsCpc\n nativeAdsTracking\n sponsoredAdsData {\n isDsaCompliant\n legalEntityName\n sponsoredAdsDesign\n __typename\n }\n __typename\n }\n policies {\n showFreeCancellation\n showNoPrepayment\n enableJapaneseUsersSpecialCase\n __typename\n }\n ribbon {\n ribbonType\n text\n __typename\n }\n recommendedDate {\n checkin\n checkout\n lengthOfStay\n __typename\n }\n showGeniusLoginMessage\n hostTraderLabel\n soldOutInfo {\n isSoldOut\n messages {\n text\n __typename\n }\n alternativeDatesMessages {\n text\n __typename\n }\n __typename\n }\n nbWishlists\n visibilityBoosterEnabled\n showAdLabel\n isNewlyOpened\n propertySustainability {\n isSustainable\n tier {\n type\n __typename\n }\n facilities {\n id\n __typename\n }\n certifications {\n name\n __typename\n }\n chainProgrammes {\n chainName\n programmeName\n __typename\n }\n levelId\n __typename\n }\n seoThemes {\n caption\n __typename\n }\n relocationMode {\n distanceToCityCenterKm\n distanceToCityCenterMiles\n distanceToOriginalHotelKm\n distanceToOriginalHotelMiles\n phoneNumber\n __typename\n }\n bundleRatesAvailable\n __typename\n}\n\nfragment Banner on Banner {\n name\n type\n isDismissible\n showAfterDismissedDuration\n position\n requestAlternativeDates\n merchId\n title {\n text\n __typename\n }\n imageUrl\n paragraphs {\n text\n __typename\n }\n metadata {\n key\n value\n __typename\n }\n pendingReviewInfo {\n propertyPhoto {\n lowResUrl {\n relativeUrl\n __typename\n }\n lowResJpegUrl {\n relativeUrl\n __typename\n }\n __typename\n }\n propertyName\n urlAccessCode\n __typename\n }\n nbDeals\n primaryAction {\n text {\n text\n __typename\n }\n action {\n name\n context {\n key\n value\n __typename\n }\n __typename\n }\n __typename\n }\n secondaryAction {\n text {\n text\n __typename\n }\n action {\n name\n context {\n key\n value\n __typename\n }\n __typename\n }\n __typename\n }\n iconName\n flexibleFilterOptions {\n optionId\n filterName\n __typename\n }\n trackOnView {\n type\n experimentHash\n value\n __typename\n }\n dateFlexQueryOptions {\n text {\n text\n __typename\n }\n action {\n name\n context {\n key\n value\n __typename\n }\n __typename\n }\n isApplied\n __typename\n }\n __typename\n}\n\nfragment Carousel on Carousel {\n aggregatedCountsByFilterId\n carouselId\n position\n contentType\n hotelId\n name\n soldoutProperties\n priority\n themeId\n frontierThemeIds\n title {\n text\n __typename\n }\n slides {\n captionText {\n text\n __typename\n }\n name\n photoUrl\n subtitle {\n text\n __typename\n }\n type\n title {\n text\n __typename\n }\n action {\n context {\n key\n value\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n}\n\nfragment DestinationLocation on DestinationLocation {\n name {\n text\n __typename\n }\n inName {\n text\n __typename\n }\n countryCode\n ufi\n __typename\n}\n\nfragment FilterData on Filter {\n trackOnView {\n type\n experimentHash\n value\n __typename\n }\n trackOnClick {\n type\n experimentHash\n value\n __typename\n }\n name\n field\n category\n filterStyle\n title {\n text\n translationTag {\n translation\n __typename\n }\n __typename\n }\n subtitle\n options {\n parentId\n genericId\n trackOnView {\n type\n experimentHash\n value\n __typename\n }\n trackOnClick {\n type\n experimentHash\n value\n __typename\n }\n trackOnSelect {\n type\n experimentHash\n value\n __typename\n }\n trackOnDeSelect {\n type\n experimentHash\n value\n __typename\n }\n trackOnViewPopular {\n type\n experimentHash\n value\n __typename\n }\n trackOnClickPopular {\n type\n experimentHash\n value\n __typename\n }\n trackOnSelectPopular {\n type\n experimentHash\n value\n __typename\n }\n trackOnDeSelectPopular {\n type\n experimentHash\n value\n __typename\n }\n ...FilterOption\n __typename\n }\n filterLayout {\n isCollapsable\n collapsedCount\n __typename\n }\n stepperOptions {\n min\n max\n default\n selected\n title {\n text\n translationTag {\n translation\n __typename\n }\n __typename\n }\n field\n labels {\n text\n translationTag {\n translation\n __typename\n }\n __typename\n }\n trackOnView {\n type\n experimentHash\n value\n __typename\n }\n trackOnClick {\n type\n experimentHash\n value\n __typename\n }\n trackOnSelect {\n type\n experimentHash\n value\n __typename\n }\n trackOnDeSelect {\n type\n experimentHash\n value\n __typename\n }\n trackOnClickDecrease {\n type\n experimentHash\n value\n __typename\n }\n trackOnClickIncrease {\n type\n experimentHash\n value\n __typename\n }\n trackOnDecrease {\n type\n experimentHash\n value\n __typename\n }\n trackOnIncrease {\n type\n experimentHash\n value\n __typename\n }\n __typename\n }\n sliderOptions {\n min\n max\n minSelected\n maxSelected\n minPriceStep\n minSelectedFormatted\n currency\n histogram\n selectedRange {\n translation\n __typename\n }\n __typename\n }\n __typename\n}\n\nfragment FilterOption on Option {\n optionId: id\n count\n selected\n urlId\n source\n additionalLabel {\n text\n translationTag {\n translation\n __typename\n }\n __typename\n }\n value {\n text\n translationTag {\n translation\n __typename\n }\n __typename\n }\n starRating {\n value\n symbol\n caption {\n translation\n __typename\n }\n showAdditionalInfoIcon\n __typename\n }\n __typename\n}\n\nfragment LandingPageBreadcrumb on LandingPageBreadcrumb {\n destType\n name\n urlParts\n __typename\n}\n\nfragment MatchingUnitConfigurations on SearchResultProperty {\n matchingUnitConfigurations {\n commonConfiguration {\n name\n unitId\n bedConfigurations {\n beds {\n count\n type\n __typename\n }\n nbAllBeds\n __typename\n }\n nbAllBeds\n nbBathrooms\n nbBedrooms\n nbKitchens\n nbLivingrooms\n nbUnits\n unitTypeNames {\n translation\n __typename\n }\n localizedArea {\n localizedArea\n unit\n __typename\n }\n __typename\n }\n unitConfigurations {\n name\n unitId\n bedConfigurations {\n beds {\n count\n type\n __typename\n }\n nbAllBeds\n __typename\n }\n apartmentRooms {\n config {\n roomId: id\n roomType\n bedTypeId\n bedCount: count\n __typename\n }\n roomName: tag {\n tag\n translation\n __typename\n }\n __typename\n }\n nbAllBeds\n nbBathrooms\n nbBedrooms\n nbKitchens\n nbLivingrooms\n nbUnits\n unitTypeNames {\n translation\n __typename\n }\n localizedArea {\n localizedArea\n unit\n __typename\n }\n unitTypeId\n __typename\n }\n __typename\n }\n __typename\n}\n\nfragment PropertyBlocks on SearchResultProperty {\n blocks {\n blockId {\n roomId\n occupancy\n policyGroupId\n packageId\n mealPlanId\n bundleId\n __typename\n }\n finalPrice {\n amount\n currency\n __typename\n }\n originalPrice {\n amount\n currency\n __typename\n }\n onlyXLeftMessage {\n tag\n variables {\n key\n value\n __typename\n }\n translation\n __typename\n }\n freeCancellationUntil\n hasCrib\n blockMatchTags {\n childStaysForFree\n __typename\n }\n thirdPartyInventoryContext {\n isTpiBlock\n __typename\n }\n __typename\n }\n __typename\n}\n\nfragment PriceDisplayInfoIrene on PriceDisplayInfoIrene {\n badges {\n name {\n translation\n __typename\n }\n tooltip {\n translation\n __typename\n }\n style\n identifier\n __typename\n }\n chargesInfo {\n translation\n __typename\n }\n displayPrice {\n copy {\n translation\n __typename\n }\n amountPerStay {\n amount\n amountRounded\n amountUnformatted\n currency\n __typename\n }\n __typename\n }\n priceBeforeDiscount {\n copy {\n translation\n __typename\n }\n amountPerStay {\n amount\n amountRounded\n amountUnformatted\n currency\n __typename\n }\n __typename\n }\n rewards {\n rewardsList {\n termsAndConditions\n amountPerStay {\n amount\n amountRounded\n amountUnformatted\n currency\n __typename\n }\n breakdown {\n productType\n amountPerStay {\n amount\n amountRounded\n amountUnformatted\n currency\n __typename\n }\n __typename\n }\n __typename\n }\n rewardsAggregated {\n amountPerStay {\n amount\n amountRounded\n amountUnformatted\n currency\n __typename\n }\n copy {\n translation\n __typename\n }\n __typename\n }\n __typename\n }\n useRoundedAmount\n discounts {\n amount {\n amount\n amountRounded\n amountUnformatted\n currency\n __typename\n }\n name {\n translation\n __typename\n }\n description {\n translation\n __typename\n }\n itemType\n productId\n __typename\n }\n excludedCharges {\n excludeChargesAggregated {\n copy {\n translation\n __typename\n }\n amountPerStay {\n amount\n amountRounded\n amountUnformatted\n currency\n __typename\n }\n __typename\n }\n excludeChargesList {\n chargeMode\n chargeInclusion\n chargeType\n amountPerStay {\n amount\n amountRounded\n amountUnformatted\n currency\n __typename\n }\n __typename\n }\n __typename\n }\n taxExceptions {\n shortDescription {\n translation\n __typename\n }\n longDescription {\n translation\n __typename\n }\n __typename\n }\n __typename\n}\n\nfragment BookerExperienceData on SearchResultProperty {\n bookerExperienceContentUIComponentProps {\n ... on BookerExperienceContentLoyaltyBadgeListProps {\n badges {\n variant\n key\n title\n popover\n logoSrc\n logoAlt\n __typename\n }\n __typename\n }\n ... on BookerExperienceContentFinancialBadgeProps {\n paymentMethod\n backgroundColor\n hideAccepted\n __typename\n }\n __typename\n }\n __typename\n}\n\nfragment SearchMetadata on SearchMeta {\n availabilityInfo {\n hasLowAvailability\n unavailabilityPercent\n totalAvailableNotAutoextended\n __typename\n }\n boundingBoxes {\n swLat\n swLon\n neLat\n neLon\n type\n __typename\n }\n childrenAges\n dates {\n checkin\n checkout\n lengthOfStayInDays\n __typename\n }\n destId\n destType\n guessedLocation {\n destId\n destType\n destName\n __typename\n }\n maxLengthOfStayInDays\n nbRooms\n nbAdults\n nbChildren\n userHasSelectedFilters\n customerValueStatus\n isAffiliateBookingOwned\n affiliatePartnerChannelId\n affiliateVerticalType\n geniusLevel\n __typename\n}\n\nfragment SearchResultsBreadcrumb on SearchResultsBreadcrumb {\n destId\n destType\n name\n __typename\n}\n\nfragment SorterFields on SorterOption {\n type: name\n captionTranslationTag {\n translation\n __typename\n }\n tooltipTranslationTag {\n translation\n __typename\n }\n isSelected: selected\n __typename\n}\n\nfragment TripTypesData on TripTypes {\n beach {\n isBeachUfi\n isEnabledBeachUfi\n __typename\n }\n ski {\n isSkiExperience\n isSkiScaleUfi\n __typename\n }\n __typename\n}\n\nfragment ZeroResultsSection on ZeroResultsSection {\n title {\n text\n __typename\n }\n primaryAction {\n text {\n text\n __typename\n }\n action {\n name\n __typename\n }\n __typename\n }\n paragraphs {\n text\n __typename\n }\n type\n __typename\n}\n\nfragment PreviousSearches on PreviousSearch {\n childrenAges\n __typename\n}\n\nfragment FrontierThemes on FrontierTheme {\n id\n name\n selected\n __typename\n}\n\nfragment MerchRegionIrene on MerchComponentsResultIrene {\n regions {\n id\n components {\n ... on PromotionalBannerIrene {\n promotionalBannerCampaignId\n contentArea {\n title {\n ... on PromotionalBannerSimpleTitleIrene {\n value\n __typename\n }\n __typename\n }\n subTitle {\n ... on PromotionalBannerSimpleSubTitleIrene {\n value\n __typename\n }\n __typename\n }\n caption {\n ... on PromotionalBannerSimpleCaptionIrene {\n value\n __typename\n }\n ... on PromotionalBannerCountdownCaptionIrene {\n campaignEnd\n __typename\n }\n __typename\n }\n buttons {\n variant\n cta {\n ariaLabel\n text\n targetLanding {\n ... on OpenContextSheet {\n sheet {\n ... on WebContextSheet {\n title\n body {\n items {\n ... on ContextSheetTextItem {\n text\n __typename\n }\n ... on ContextSheetList {\n items {\n text\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n buttons {\n variant\n cta {\n text\n ariaLabel\n targetLanding {\n ... on DirectLinkLanding {\n urlPath\n queryParams {\n name\n value\n __typename\n }\n __typename\n }\n ... on LoginLanding {\n stub\n __typename\n }\n ... on DeeplinkLanding {\n urlPath\n queryParams {\n name\n value\n __typename\n }\n __typename\n }\n ... on ResolvedLinkLanding {\n url\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n ... on SearchResultsLandingIrene {\n destType\n destId\n checkin\n checkout\n nrAdults\n nrChildren\n childrenAges\n nrRooms\n filters {\n name\n value\n __typename\n }\n __typename\n }\n ... on DirectLinkLandingIrene {\n urlPath\n queryParams {\n name\n value\n __typename\n }\n __typename\n }\n ... on LoginLandingIrene {\n stub\n __typename\n }\n ... on DeeplinkLandingIrene {\n urlPath\n queryParams {\n name\n value\n __typename\n }\n __typename\n }\n ... on SorterLandingIrene {\n sorterName\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n designVariant {\n ... on DesktopPromotionalFullBleedImageIrene {\n image: image {\n id\n url(width: 814, height: 138)\n alt\n overlayGradient\n primaryColorHex\n __typename\n }\n colorScheme\n signature\n __typename\n }\n ... on DesktopPromotionalImageLeftIrene {\n imageOpt: image {\n id\n url(width: 248, height: 248)\n alt\n overlayGradient\n primaryColorHex\n __typename\n }\n colorScheme\n signature\n __typename\n }\n ... on DesktopPromotionalImageRightIrene {\n imageOpt: image {\n id\n url(width: 248, height: 248)\n alt\n overlayGradient\n primaryColorHex\n __typename\n }\n colorScheme\n signature\n __typename\n }\n ... on MdotPromotionalFullBleedImageIrene {\n image: image {\n id\n url(width: 358, height: 136)\n alt\n overlayGradient\n primaryColorHex\n __typename\n }\n colorScheme\n signature\n __typename\n }\n ... on MdotPromotionalImageLeftIrene {\n imageOpt: image {\n id\n url(width: 128, height: 128)\n alt\n overlayGradient\n primaryColorHex\n __typename\n }\n colorScheme\n signature\n __typename\n }\n ... on MdotPromotionalImageRightIrene {\n imageOpt: image {\n id\n url(width: 128, height: 128)\n alt\n overlayGradient\n primaryColorHex\n __typename\n }\n colorScheme\n signature\n __typename\n }\n ... on MdotPromotionalImageTopIrene {\n imageOpt: image {\n id\n url(width: 128, height: 128)\n alt\n overlayGradient\n primaryColorHex\n __typename\n }\n colorScheme\n signature\n __typename\n }\n ... on MdotPromotionalIllustrationLeftIrene {\n imageOpt: image {\n id\n url(width: 200, height: 200)\n alt\n overlayGradient\n primaryColorHex\n __typename\n }\n colorScheme\n signature\n __typename\n }\n ... on MdotPromotionalIllustrationRightIrene {\n imageOpt: image {\n id\n url(width: 200, height: 200)\n alt\n overlayGradient\n primaryColorHex\n __typename\n }\n colorScheme\n signature\n __typename\n }\n __typename\n }\n __typename\n }\n ... on MerchCarouselIrene @include(if: $carouselLowCodeExp) {\n carouselCampaignId\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n}\n"
}
def generate_graphql_request(url_params: str, body: Dict, offset: int):
"""create a scrape config for the search graphql request"""
body["variables"]["input"]["pagination"]["offset"] = offset
return ScrapeConfig(
"https://www.booking.com/dml/graphql?" + url_params,
headers={
"accept":"*/*",
"cache-control":"no-cache",
"content-type": "application/json",
"origin":"https://www.booking.com",
"pragma":"no-cache",
"priority":"u=1, i",
"referer":"https://www.booking.com/searchresults.en-gb.html?" + url_params,
},
data=body,
method="POST",
asp=True
)
async def search_location_suggestions(query: str) -> Dict:
"""scrape booking.com location suggestions to find location details for search scraping"""
result = await SCRAPFLY.async_scrape(
ScrapeConfig(
url="https://accommodations.booking.com/autocomplete.json",
method="POST",
headers={
"Origin": "https://www.booking.com",
"Referer": "https://www.booking.com/",
"Content-Type": "text/plain;charset=UTF-8",
},
body=f'{{"query":"{query}","pageview_id":"","aid":800210,"language":"en-us","size":5}}',
)
)
data = json.loads(result.content)
return data
def parse_graphql_response(response: ScrapeApiResponse) -> List[Dict]:
"""parse the search results from the graphql response"""
data = json.loads(response.content)
parsed_data = data["data"]["searchQueries"]["search"]["results"]
return parsed_data
async def scrape_search(
query,
checkin: str = "", # e.g. 2023-05-30
checkout: str = "", # e.g. 2023-06-26
number_of_rooms=1,
max_pages: Optional[int] = None,
) -> List[Dict]:
"""Scrape booking.com search"""
print(f"scraping search for {query} {checkin}-{checkout}")
# first we must find destination details from provided query
# for that scrape suggestions from booking.com autocomplete and take the first one
location_suggestions = await search_location_suggestions(query)
destination = location_suggestions["results"][0]
url_params = urlencode(
{
"ss": destination["value"],
"ssne": destination["value"],
"ssne_untouched": destination["value"],
"checkin": checkin,
"checkout": checkout,
"no_rooms": number_of_rooms,
"dest_id": destination["dest_id"],
"dest_type": destination["dest_type"],
"efdco": 1,
"group_adults": 1,
"group_children": 0,
"lang": "en-gb",
"sb": 1,
"sb_travel_purpose": "leisure",
"src": "index",
"src_elem": "sb",
}
)
search_url = "https://www.booking.com/searchresults.en-gb.html?" + url_params
# first scrape the first page and find total amount of pages
first_page = await SCRAPFLY.async_scrape(ScrapeConfig(search_url, **BASE_CONFIG))
_total_results = int(first_page.selector.xpath("//h1[contains(text(),'properties found')]").re(r"([\d,]+) properties found")[0].replace(",", ""))
_max_scrape_results = max_pages * 25
if _max_scrape_results and _max_scrape_results < _total_results:
_total_results = _max_scrape_results
data = []
body = retrieve_graphql_body(first_page)
to_scrape = [
generate_graphql_request(url_params, body, offset)
for offset in range(0, _total_results, 25)
]
print(f"scraping search results from the graphql api: {len(to_scrape)} pages to request")
async for response in SCRAPFLY.concurrent_scrape(to_scrape):
try:
data.extend(parse_graphql_response(response))
except Exception as e:
print(f"Failed to parse search results: {e}")
print(f"scraped {len(data)} results from search pages")
return data
async def run():
TODAY = datetime.now().strftime('%Y-%m-%d')
WEEK_FROM_NOW = (datetime.now() + timedelta(days=7)).strftime('%Y-%m-%d')
result_search = await scrape_search(
query="Malta",
checkin=TODAY,
checkout=WEEK_FROM_NOW,
max_pages=2
)
# save the results to a json file
with open("search.json", "w", encoding="utf-8") as file:
json.dump(result_search, file, indent=2, ensure_ascii=False)
if __name__ == "__main__":
asyncio.run(run())First, we define our scrape_search() function which iterates through our previously defined search functionality to scrape all pages instead of just the first one. We accomplish this by leveraging a common web scraping pattern for handling known-size pagination — scraping the first page, determining the total number of results, and then scraping the remaining pages concurrently.
Another crucial parameter is offset which controls search result pagination. Providing an offset specifies that we want 25 results starting from a specific point in the result set. Let’s utilize this to implement comprehensive paging and hotel preview data parsing.
Additionally, we define three functions, let’s examine them:
- generate_graphql_request: Creates a request object with the required body, offset, and headers. It represents the primary search API calls for retrieving search data.
- retrieve_graphql_body: Parses the GraphQL request body from the HTML. Note that the body utilizes a query object, which can be retrieved from browser XHR calls.
- parse_graphql_response: Parses the response from the search API requests.
Here’s a sample output of our results:
Example output:
We’ve successfully scraped Booking.com’s search pages to discover hotels located in London. Furthermore, we obtained valuable metadata and URLs to the hotel pages themselves, so next we can scrape detailed hotel data and pricing information!
Scraping Booking.com Hotel Data
Now that we have a scraper capable of extracting Booking.com’s hotel preview data, we can further collect remaining hotel information like descriptions, addresses, feature lists, etc. by scraping each individual hotel URL.
To scrape Booking hotel data, we’ll directly parse the HTML using CSS and XPath selectors:
import os
import re
import json
import asyncio
from uuid import uuid4
from collections import defaultdict
from datetime import datetime, timedelta
from typing import Dict, List
from scrapfly import ScrapeApiResponse, ScrapeConfig, ScrapflyClient
BASE_CONFIG = {
"asp": True,
"country": "US",
}
SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"])
def parse_hotel(result: ScrapeApiResponse) -> Dict:
print(f"parsing hotel page: {result.context['url']}")
sel = result.selector
features = defaultdict(list)
for box in sel.xpath('//div[@data-testid="property-section--content"]/div[2]/div'):
type_ = box.xpath('.//span[contains(@data-testid, "facility-group-icon")]/../text()').get()
if not type_:
continue
feats = [f.strip() for f in box.css("li ::text").getall() if f.strip()]
features[type_] = feats
css = lambda selector, sep="": sep.join(sel.css(selector).getall()).strip()
xpath = lambda selector, sep="": sep.join(sel.xpath(selector).getall()).strip()
lat, lng = sel.css(".show_map_hp_link::attr(data-atlas-latlng)").get("0,0").split(",")
id = re.findall(r"b_hotel_id:\s*'(.+?)'", result.content)
data = {
"url": result.context["url"],
"id": id[0] if id else None,
"title": sel.css("h2::text").get(),
"description": css('[data-capla-component-boundary="b-property-web-property-page/PropertyDescriptionDesktop"] ::text', "\n"),
"address": xpath("//div[@data-testid='PropertyHeaderAddressDesktop-wrapper']//button/div/text()"),
"images": sel.css("#photo_wrapper img::attr(src)").getall(),
"lat": lat,
"lng": lng,
"features": dict(features),
}
return data
async def scrape_hotel(url: str, checkin: str, price_n_days=61) -> Dict:
"""
Scrape Booking.com hotel data and pricing information.
"""
print(f"scraping hotel {url} {checkin} with {price_n_days} days of pricing data")
session = str(uuid4()).replace("-", "")
result = await SCRAPFLY.async_scrape(
ScrapeConfig(
url,
session=session,
**BASE_CONFIG,
)
)
hotel = parse_hotel(result)
# To scrape price we'll be calling Booking.com's graphql service
# in particular we'll be calling AvailabilityCalendar query
# first, extract hotel variables:
_hotel_country = re.findall(r'hotelCountry:\s*"(.+?)"', result.content)[0]
_hotel_name = re.findall(r'hotelName:\s*"(.+?)"', result.content)[0]
_csrf_token = re.findall(r"b_csrf_token:\s*'(.+?)'", result.content)[0]
# then create graphql query
gql_body = json.dumps(
{
"operationName": "AvailabilityCalendar",
# hotel varialbes go here
# you can adjust number of adults, room number etc.
"variables": {
"input": {
"travelPurpose": 2,
"pagenameDetails": {
"countryCode": _hotel_country,
"pagename": _hotel_name,
},
"searchConfig": {
"searchConfigDate": {
"startDate": checkin,
"amountOfDays": price_n_days,
},
"nbAdults": 2,
"nbRooms": 1,
},
}
},
"extensions": {},
# this is the query itself, don't alter it
"query": "query AvailabilityCalendar($input: AvailabilityCalendarQueryInput!) {\n availabilityCalendar(input: $input) {\n ... on AvailabilityCalendarQueryResult {\n hotelId\n days {\n available\n avgPriceFormatted\n checkin\n minLengthOfStay\n __typename\n }\n __typename\n }\n ... on AvailabilityCalendarQueryError {\n message\n __typename\n }\n __typename\n }\n}\n",
},
# note: this removes unnecessary whitespace in JSON output
separators=(",", ":"),
)
# scrape booking graphql
result_price = await SCRAPFLY.async_scrape(
ScrapeConfig(
"https://www.booking.com/dml/graphql?lang=en-gb",
method="POST",
body=gql_body,
session=session,
# note that we need to set headers to avoid being blocked
headers={
"content-type": "application/json",
"x-booking-csrf-token": _csrf_token,
"referer": result.context["url"],
"origin": "https://www.booking.com",
},
**BASE_CONFIG,
)
)
price_data = json.loads(result_price.content)
hotel["price"] = price_data["data"]["availabilityCalendar"]["days"]
return hotel
async def run():
WEEK_FROM_NOW = (datetime.now() + timedelta(days=7)).strftime('%Y-%m-%d')
result_hotel = await scrape_hotel(
"https://www.booking.com/hotel/gb/gardencourthotel.en-gb.html",
checkin=WEEK_FROM_NOW,
price_n_days=7,
)
# save the results to a json file
with open("hotel.json", "w", encoding="utf-8") as file:
json.dump(result_hotel, file, indent=2, ensure_ascii=False)
if __name__ == "__main__":
asyncio.run(run())Here, we establish our hotel page scraping functionality. Our scrape_hotel function accepts a list of hotel URLs which we scrape via simple GET requests for HTML data. We then employ our HTML parsing library to extract hotel information using selectors.
Example output:
There’s significantly more data available on each page, but to maintain tutorial brevity we focused on a few example fields only.
Scraping Booking.com Hotel Reviews
To scrape Booking.com hotel reviews, let’s examine what occurs when we explore the reviews page. Let’s click page 2 and observe what happens in our browser’s web inspector (F12 in major browsers):
We can observe a background request being made when we click the page 2 link
Similar to product search, product reviews can be extracted from the GraphQL endpoint:
import os
import re
import json
import asyncio
from uuid import uuid4
from typing import Dict, List, Optional
from scrapfly import ScrapeApiResponse, ScrapeConfig, ScrapflyClient
BASE_CONFIG = {
"asp": True,
"country": "US",
}
SCRAPFLY = ScrapflyClient(key=os.environ["SCRAPFLY_KEY"])
def retrieve_reviews_api_xhr_call(result: ScrapeApiResponse) -> Dict:
"""retrieve the reviews xhr call from the captured browser data"""
_xhr_calls = result.scrape_result["browser_data"]["xhr_call"]
for xhr in _xhr_calls:
if "reviewCard" in xhr["response"]["body"]:
return xhr
async def scrape_hotel_reviews(url: str, max_pages: Optional[int] = None) -> List[Dict]:
"""scrape hotel review data"""
reviews_data = []
reviews_page_url = url + "?force_referer=#tab-reviews"
session_id = str(uuid4()).replace("-", "")
print(f"scraping the main reviews page for the url {url} before scraping the graphql api")
main_reviews_page = await SCRAPFLY.async_scrape(
ScrapeConfig(reviews_page_url, **BASE_CONFIG, render_js=True, rendering_wait=5000, session=session_id)
)
reviews_xhr_call = retrieve_reviews_api_xhr_call(main_reviews_page)
gql_body = json.loads(reviews_xhr_call["body"])
total_review_count = int(json.loads(reviews_xhr_call["response"]["body"])["data"]["reviewListFrontend"]["reviewsCount"])
total_review_pages = math.ceil(total_review_count / 10)
_csrf_token = re.findall(r"b_csrf_token:\s*'(.+?)'", main_reviews_page.content)[0]
if max_pages is None:
max_pages = total_review_pages
if max_pages is not None and max_pages > total_review_pages:
max_pages = total_review_pages
print(f"scraping {max_pages} review pages concurrently using the graphql api")
def update_gql_body(gql_body: Dict, offset: int) -> Dict:
gql_body['variables']['input']['skip'] = offset
return gql_body
remaining_pages = [
ScrapeConfig(
"https://www.booking.com/dml/graphql?lang=en-gb",
method="POST",
body=json.dumps(update_gql_body(gql_body, offset)),
session=session_id,
# note that we need to set headers to avoid being blocked
headers={
"content-type": "application/json",
"x-booking-csrf-token": _csrf_token,
"referer": main_reviews_page.context["url"],
"origin": "https://www.booking.com",
},
**BASE_CONFIG,
)
for offset in range(0, max_pages * 10, 10)
]
async for response in SCRAPFLY.concurrent_scrape(remaining_pages):
reviews_data.extend(json.loads(response.content)["data"]["reviewListFrontend"]["reviewCard"])
print(f"scraped {len(reviews_data)} reviews from the hotel reviews api for the url {url}")
return reviews_data
async def main():
reviews_data = await scrape_hotel_reviews("https://www.booking.com/hotel/gb/gardencourthotel.en-gb.html", max_pages=3)
# save the results to a json file
with open("hotel_reviews.json", "w", encoding="utf-8") as file:
json.dump(reviews_data, file, indent=2, ensure_ascii=False)
if __name__ == "__main__":
asyncio.run(main())In our scraper code above, we utilize what we learned previously: we collect the first page to extract the total number of pages and then scrape the remaining pages concurrently.
Here’s what the retrieved Booking hotel review data should look like:
Example output:
Finally – our scraper can discover hotels, extract hotel preview data and then scrape each hotel page for hotel information, pricing data and reviews!
However, to adopt this scraper at scale we need one final component – web scraper blocking avoidance.
Bypass Blocking and Captchas with Webparsers
We examined how to scrape Booking.com, though unfortunately when scraping at scale it’s very likely that Booking will start to either block us or serve us captchas, which will hinder or completely disable our web scraper.
Webparsers provides web scraping, screenshot, and extraction APIs for data collection at scale.
- Anti-bot protection bypass – scrape web pages without blocking!
- Rotating residential proxies – prevent IP address and geographic blocks.
- JavaScript