Scraping Rightmove reveals extensive property data including prices, coordinates, agent details, and media content that powers market research, investment analysis, and proptech applications. Whether tracking market trends or building automated property discovery tools, extracting this data provides crucial market insights.
We’ll extract real estate information for properties listed for sale or rent throughout the UK using Python combined with several community libraries.
Additionally, we’ll demonstrate how to discover listings through Rightmove’s search functionality, enabling you to monitor new properties and gain real-time market advantages.
Key Takeaways
Develop robust rightmove scrapers using Python with httpx and parsel libraries, extracting comprehensive UK property data from hidden web sources while managing anti-blocking measures for thorough real estate data collection.
- Access RightMove’s backend APIs for UK property listings and rental information without JavaScript rendering dependencies
- Process JSON responses to extract complete real estate data including pricing, locations, and property specifications
- Execute PAGE_MODEL extraction from JavaScript variables using XPath selectors and JSON parsing methods
- Manage RightMove’s anti-scraping protections with appropriate headers and request timing for UK market data acquisition
- Extract organized property information including listing details, agent data, and property characteristics
- Set up property search and monitoring systems to track new listings and market developments
What is Rightmove?
Rightmove stands as the UK’s most popular property marketplace, consolidating listings from thousands of estate agents and developers across England, Scotland, Wales, and Northern Ireland. It serves as the primary destination for British property seekers looking to rent, purchase, or monitor housing prices.
For international audiences, considering Rightmove as the “UK equivalent of Zillow” provides useful context. Scraping this platform delivers near real-time insights into regional supply patterns, price movements, inventory levels, and agent performance metrics, proving invaluable for institutional analysts, proptech platforms, and independent developers.
Why Scrape RightMove.co.uk?
Rightmove transcends a simple classifieds platform—it represents the living heartbeat of the UK housing market, receiving minute-by-minute updates from thousands of agencies. Scraping enables you to:
- Measure supply and demand across micro-markets well before official statistics are released.
- Track price reductions, days-on-market, and EPC ratings for investment research purposes.
- Automatically populate proptech products with fresh listings, media content, and agent information.
Since Rightmove handles both rental and sales properties, you can leverage identical scraping techniques for buy-to-let analysis, relocation platforms, or enterprise-grade comparable studies. Explore related targets like Zillow and Realtor.com for broader market coverage.
What Rightmove Data Can You Scrape?
While Rightmove’s frontend may appear restrictive, the underlying data payloads are remarkably comprehensive. Here’s what we typically extract:
Property Details: Price history, bedroom/bathroom counts, tenure information, transaction types, EPC ratings, and marketing descriptors. Ideal for comparables, affordability modeling, and trend analysis.
Address and Location: Complete display addresses plus latitude/longitude coordinates, postcode breakdowns, and neighborhood identifiers that facilitate straightforward geospatial analysis.
Images and Media: High-resolution photographs, floor plans, 3D virtual tours, and brochure PDFs. Perfect for enriching internal property databases.
Agent and Agency Information: Branch names, contact numbers, and company affiliations. Supports CRM enhancement, competitive analysis, and outreach automation.
Review this sample dataset demonstrating our scraping capabilities:
Scrape Result Preview
As demonstrated, RightMove contains extensive valuable data fields. Let’s examine how to extract them effectively!
Why Rightmove Scraping Fails Without Proper Tools
Rightmove actively resists basic scraping attempts. Common obstacles include:
Rate limiting: Excessive requests from single IP addresses result in silent response throttling or empty data returns.
Session validation: Certain listing pages require active cookie sessions or tokens that expire rapidly.
Dynamic HTML & hidden JSON: Property information exists within JavaScript objects or XHR responses, making simple HTML scraping ineffective.
Custom scraper development involves managing IP rotation, session persistence, headless rendering, and specialized JSON parsing. Webparsers automates these complexities through residential/mobile proxies, session management, JavaScript rendering, and structured response processing, allowing focus on data extraction itself.
Project Setup
This tutorial utilizes Python with three community packages:
httpx – HTTP client library enabling communication with RightMove.co.uk servers
parsel – HTML parsing library for processing scraped HTML files
jmespath – JSON parsing library for extracting details from large JSON datasets
This scraper primarily works with JSON and hidden web data, so we’ll mainly use httpx and jmespath packages.
Install all packages easily via pip:
$ pip install httpx parsel jmespath
Alternatively, substitute httpx with other HTTP clients like requests, as we only need basic HTTP functionality that’s nearly identical across libraries. For parsing, beautifulsoup serves as an excellent parsel alternative.
We’ll use jmespath for parsing, covered comprehensively in our parsing JSON with JMESPath tutorial.
🧙♂️ We’ll also provide Webparsers SDK versions of each code snippet for users utilizing Webparsers for scraping operations.
How Rightmove Data is Loaded and How to Scrape it
Understanding Rightmove’s browser data rendering process is crucial before coding.
Hidden JSON in Script Tags
Individual property pages embed PAGE_MODEL JavaScript variables containing comprehensive data: pricing, addresses, images, agent information, EPC data, and more. The process involves:
- Request property pages with browser-like headers.
- Parse HTML and locate
<script>elements assigning PAGE_MODEL. - Extract JSON content, load with
json.loads, and select relevant fields.
This represents classic hidden-data scraping: payloads never appear in rendered DOM but are fully exposed in source code.
XHR Calls for Search and List Pages
Search results, pagination, and “similar listings” widgets utilize Rightmove’s internal REST endpoints (e.g., /api/_search). To capture them:
- Initiate searches (or mimic requests) with proper query parameters like
locationIdentifier,index, andnumberOfPropertiesPerPage. - Capture JSON responses directly—no HTML parsing required.
- Iterate through properties, collecting IDs, URLs, and pagination hints for continuation.
This approach is called hidden-API scraping. Once endpoints are identified, you can efficiently query across regions with minimal requests.
Scraping Rightmove Property Data
Let’s begin our scraper by examining single listing property data extraction.
RightMove employs JSON-powered frontend rendering. Scraping such pages is termed hidden web data scraping since data resides in JavaScript variables rather than HTML source code.
First, navigate to any RightMove property page. To locate hidden data, select unique data values like description portions and search HTML source code. For example, searching “rare opportunity” in listing HTML source reveals data stored in the PAGE_MODEL HTML variable:
page source of rightmove property listing
Our scraping approach involves:
- Retrieve property page HTML using httpx
- Use parsel to parse HTML and locate
<script>elements containing PAGE_MODEL variables - Load PAGE_MODEL JSON as Python dictionary and process it
Let’s extract this data using httpx and parsel packages:
<div>
<h4>Python</h4>
<pre><code>import asyncio
import json
from typing import List
from httpx import AsyncClient, Response
from parsel import Selector
# 1. establish HTTP client with browser-like headers to avoid being blocked
client = AsyncClient(
headers={
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9,lt;q=0.8,et;q=0.7,de;q=0.6",
},
follow_redirects=True,
http2=True, # enable http2 to reduce block chance
timeout=30,
)
# XXX: we'll fill this in later
def parse_property(data):
"""parse rightmove property data to only necessary fields"""
return data
def find_json_objects(text: str, decoder=json.JSONDecoder()):
"""Find JSON objects in text, and generate decoded JSON data"""
pos = 0
while True:
match = text.find("{", pos)
if match == -1:
break
try:
result, index = decoder.raw_decode(text[match:])
yield result
pos = match + index
except ValueError:
pos = match + 1
# This function will find the PAGE_MODEL javascript variable and extract it
def extract_property(response: Response) -> dict:
"""extract property data from rightmove PAGE_MODEL javascript variable"""
selector = Selector(response.text)
data = selector.xpath("//script[contains(.,'PAGE_MODEL = ')]/text()").get()
if not data:
print(f"page {response.url} is not a property listing page")
return
json_data = list(find_json_objects(data))[0]
return json_data["propertyData"]
# this is our main scraping function that takes urls and returns the data
async def scrape_properties(urls: List[str]) -> List[dict]:
"""Scrape Rightmove property listings for property data"""
to_scrape = [client.get(url) for url in urls]
properties = []
for response in asyncio.as_completed(to_scrape):
response = await response
properties.append(parse_property(extract_property(response)))
return properties
# Eexample run:
async def run():
data = await scrape_properties([
"https://www.rightmove.co.uk/properties/149360984#/",
"https://www.rightmove.co.uk/properties/136408088#/",
"https://www.rightmove.co.uk/properties/148922639#/",
])
print(json.dumps(data, indent=2))
if __name__ == "__main__":
asyncio.run(run())</code></pre>
</div>Example Output
In our Rightmove scraper above, we established httpx.AsyncClient with browser-like headers to prevent blocking. This object enables asynchronous HTML page retrieval. We implement this in our scrape_properties function using asyncio.as_completed to gather and send HTTP requests concurrently. We then parse responses for hidden JSON data using simple XPath that locates <script> elements containing PAGE_MODEL variable names.
JSON Parsing
The output dataset is extensive and contains considerable data we don’t necessarily need. Let’s reduce it using jmespath to create more digestible results. For this, let’s complete the parse_property function defined earlier:
Quick Intro to Parsing JSON with JMESPath in Python
from typing import TypedDict
class PropertyResult(TypedDict):
"""this is what our result dataset will look like"""
id: str
available: bool
archived: bool
phone: str
bedrooms: int
bathrooms: int
type: str
property_type: str
tags: list
description: str
title: str
subtitle: str
price: str
price_sqft: str
address: dict
latitude: float
longitude: float
features: list
history: dict
photos: list
floorplans: list
agency: dict
industryAffiliations: list
nearest_airports: list
nearest_stations: list
sizings: list
brochures: list
def parse_property(data) -> PropertyResult:
"""parse rightmove cache data for proprety information"""
# here we define field name to JMESPath mapping
parse_map = {
"id": "id",
"available": "status.published",
"archived": "status.archived",
"phone": "contactInfo.telephoneNumbers.localNumber",
"bedrooms": "bedrooms",
"bathrooms": "bathrooms",
"type": "transactionType",
"property_type": "propertySubType",
"tags": "tags",
"description": "text.description",
"title": "text.pageTitle",
"subtitle": "text.propertyPhrase",
"price": "prices.primaryPrice",
"price_sqft": "prices.pricePerSqFt",
"address": "address",
"latitude": "location.latitude",
"longitude": "location.longitude",
"features": "keyFeatures",
"history": "listingHistory",
"photos": "images[*].{url: url, caption: caption}",
"floorplans": "floorplans[*].{url: url, caption: caption}",
"agency": """customer.{
id: branchId,
branch: branchName,
company: companyName,
address: displayAddress,
commercial: commercial,
buildToRent: buildToRent,
isNew: isNewHomeDeveloper
}""",
"industryAffiliations": "industryAffiliations[*].name",
"nearest_airports": "nearestAirports[*].{name: name, distance: distance}",
"nearest_stations": "nearestStations[*].{name: name, distance: distance}",
"sizings": "sizings[*].{unit: unit, min: minimumSize, max: maximumSize}",
"brochures": "brochures",
}
results = {}
for key, path in parse_map.items():
value = jmespath.search(path, data)
results[key] = value
return results
Example Output
Using JMESPath, we condensed the hidden web data dataset to essential property data fields, making it significantly easier to integrate into existing data pipelines!
Scraping Rightmove Search
Now that we understand RightMove property scraping, let’s examine how to discover them using RightMove’s search functionality.
Opening developer tools and inspecting the network tab reveals that RightMove uses a REST API for search results:
When we click next page we can see a backend API request being made
Search results are fetched from this URL using GET requests:
https://www.rightmove.co.uk/api/_search? locationIdentifier=REGION%5E61294& numberOfPropertiesPerPage=24& radius=0.0& sortType=6& index=24& includeSSTC=false& viewType=LIST& channel=BUY& areaSizeUnit=sqft& currencyCode=GBP& isFetching=false
However, scraping this search requires location identifiers which appear numeric (in this example it’s REGION^61294). How do we obtain this identifier?
Inspection reveals another API endpoint resolving location names to identifiers:
Background requests are being made to location hint API while we type
https://www.rightmove.co.uk/typeAhead/uknostreet/CO/RN/WA/LL/
This search endpoint accepts two-character segments and suggests UK location IDs. We can use this endpoint to resolve search queries to location IDs and then scrape the search API.
Let’s use these endpoints to develop our search scraper:
Python
import asyncio
import json
from typing import List, TypedDict
from urllib.parse import urlencode
from httpx import AsyncClient, Response
async def find_locations(query: str) -> List[str]:
"""use rightmove's typeahead api to find location IDs. Returns list of location IDs in most likely order"""
# rightmove uses two character long tokens so "cornwall" becomes "CO/RN/WA/LL"
tokenize_query = "".join(c + ("/" if i % 2 == 0 else "") for i, c in enumerate(query.upper(), start=1))
url = f"https://www.rightmove.co.uk/typeAhead/uknostreet/{tokenize_query.strip('/')}/"
response = await client.get(url)
data = json.loads(response.text)
return [prediction["locationIdentifier"] for prediction in data["typeAheadLocations"]]
async def scrape_search(location_id: str) -> dict:
RESULTS_PER_PAGE = 24
def make_url(offset: int) -> str:
url = "https://www.rightmove.co.uk/api/_search?"
params = {
"areaSizeUnit": "sqft",
"channel": "BUY", # BUY or RENT
"currencyCode": "GBP",
"includeSSTC": "false",
"index": offset, # page offset
"isFetching": "false",
"locationIdentifier": location_id, #e.g.: "REGION^61294",
"numberOfPropertiesPerPage": RESULTS_PER_PAGE,
"radius": "0.0",
"sortType": "6",
"viewType": "LIST",
}
return url + urlencode(params)
first_page = await client.get(make_url(0))
first_page_data = json.loads(first_page.content)
total_results = int(first_page_data['resultCount'].replace(',', ''))
results = first_page_data['properties']
other_pages = []
# rightmove sets the API limit to 1000 properties
max_api_results = 1000
for offset in range(RESULTS_PER_PAGE, total_results, RESULTS_PER_PAGE):
# stop scraping more pages when the scraper reach the API limit
if offset >= max_api_results:
break
other_pages.append(client.get(make_url(offset)))
for response in asyncio.as_completed(other_pages):
response = await response
data = json.loads(response.text)
results.extend(data['properties'])
return results
# Example run:
async def run():
cornwall_id = (await find_locations("cornwall"))[0]
print(cornwall_id)
cornwall_results = await scrape_search(cornwall_id)
print(json.dumps(cornwall_results, indent=2))
if __name__ == "__main__":
asyncio.run(run())Example Output
By scraping search functionality, we can obtain multiple property results with just a few requests. This approach is significantly more efficient than scraping individual properties, though individual property data remains more detailed and comprehensive.
We can also utilize this search scraper for new property listing notifications by sorting by listing date and continuously monitoring first-page results.
Bypass Rightmove Blocking with Webparsers
We can see that given appropriate tools and techniques, RightMove represents an accessible scraping target. However, scaling our scraper beyond a few requests risks blocking.
Webparsers middleware
Webparsers provides web scraping, screenshot, and extraction APIs for large-scale data collection.
- Anti-bot protection bypass – scrape web pages without blocking!
- Rotating residential proxies – prevent IP address and geographic blocks.
- JavaScript rendering – scrape dynamic web pages through cloud browsers.
- Full browser automation – control browsers to scroll, input and click on objects.
- Format conversion – scrape as HTML, JSON, Text, or Markdown.
- Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.
For more on scraping RightMove with Webparsers check out Full Scraper Code section
FAQs
How do I handle RightMove’s anti-bot protection when scraping at scale?
Use rotating residential or mobile proxies, implement realistic request delays (2-5 seconds), rotate user-agents and headers, use headless browsers for JavaScript rendering, and consider using anti-bot bypass services like Webparsers.
Can I scrape RightMove’s property history and price changes?
Yes, property history and price changes are usually part of the public HTML. Locate the HTML elements containing this data using CSS selectors or XPath. Be mindful of the volume and potential copyright of user-generated content.
How do I deal with pagination and infinite scrolling on RightMove’s search pages?
For pagination, identify the URL pattern for next pages. For infinite scrolling, use a headless browser to scroll down and trigger JavaScript to load more content, then extract the newly loaded data.
Why does RightMove return empty JSON data when scraping property pages?
RightMove likely uses anti-bot measures that detect automated requests. They might return empty or obfuscated data to scrapers. Use rotating proxies, realistic headers, and potentially headless browsers to bypass these protections.
Does RightMove.co.uk have a public web API?
No, there’s no public web API available for retrieving public property data from RightMove. However, it’s perfectly legal to scrape the website using Python so we data can be retrieved using a web scraper as described in this article.
What are other UK real estate websites I can scrape?
Besides RightMove, Zoopla is another major UK real estate platform that can be scraped for property data. Both sites cover the UK market comprehensively and can be used together for complete market coverage.
Summary
Web scraping on platforms like RightMove can seem intimidating, but as we’ve explored throughout this article:
- How to scrape property data from RightMove using Python
- Retrieving and parsing hidden JSON data from property pages
- Utilizing open-source libraries such as httpx, parsel, and jmespath for web scraping and data extraction
- Techniques to extract important data fields efficiently from listings
- Addressing anti-bot protections and scraping at scale
- Introduction to Webparsers’ API to help bypass blocks and session management challenges
This article gives you practical techniques and code snippets to get started. With the right tools and strategies, including solutions like Webparsers for handling anti-bot protections, web scraping can unlock valuable data from even the most protected sites.
Legal Disclaimer and Precautions
This tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect and here’s a good summary of what not to do:
- Do not scrape at rates that could damage the website.
- Do not scrape data that’s not available publicly.
- Do not store PII of EU citizens who are protected by GDPR.
- Do not repurpose the entire public datasets which can be illegal in some countries.
Webparsers does not offer legal advice but these are good general rules to follow in web scraping and for more you should consult a lawyer.