In this web scraping tutorial, we’ll be scraping idealista.com – the biggest real estate marketplace in Spain, Portugal and Italy.
In this guide, we’ll be exploring real estate data scraping by taking a look at Idealista.com. We’ll be scraping common property data points like property pricing, addresses, photos and agent phone numbers.
When it comes to web scraping, Idealista.com is a traditional scraping target. To scrape it, we’ll cover popular web scraping techniques used in Python like HTML parsing using CSS Selectors and concurrent requests using asyncio.
Finally, we’ll also cover tracking to scrape newly listed properties – giving us an upper hand in real estate discovery and bidding.
Key Takeaways
Master idealista api scraping with advanced Python techniques, real estate data extraction, and property monitoring for comprehensive market analysis.
- Reverse engineer Idealista’s API endpoints by intercepting browser network requests and analyzing JSON responses
- Extract structured property data including prices, locations, and property details from listing pages
- Implement pagination handling and search parameter management for comprehensive property data collection
- Configure proxy rotation and fingerprint management to avoid detection and rate limiting
- Use specialized tools like Webparsers for automated Idealista scraping with anti-blocking features
- Implement data validation and error handling for reliable property information extraction
In this article, we’ll focus on the Spanish version of the website (Idealista.com) though both Italian and Portuguese version function the same and our scraper code should work for these sources as well.
Why Scrape Idealista.com?
Idealista.com is one of the biggest real estate websites in Spain (as well as Italy and Portugal) making it the biggest public real estate dataset for these areas. Containing fields like real estate prices, listing locations and sale dates and general property information.
This data provides valuable insights for market analytics, housing industry research, and competitive analysis.
Project Setup
In this tutorial, we’ll be using Python with two community packages:
- httpx – HTTP client library which will let us communicate with Idealista.com’s servers
- parsel – HTML parsing library which will help us to parse our web scraped HTML files using CSS selectors and XPath selectors.
These packages can be easily installed via the pip install command:
$ pip install httpx parselAlternatively, feel free to swap httpx out with any other HTTP client package such as requests as we’ll only need basic HTTP functions which are almost interchangeable in every library. As for, parsel, another great alternative is the beautifulsoup package.
Scraping Idealista Property Data
Let’s start by taking a look at how to scrape Idealista for a single property. In later sections, we’ll also take a look at how to find any properties and scrape them using this property scraper.
For example, let’s start by examining a listing page and where all of the information is stored on it. Let’s select a random property listing, like:
idealista.com/en/inmueble/94156485/
For parsing data on Idealista, we’ll be using CSS selectors, so let’s identify the fields we want to scrape:
screenshot and markup of idealista property page
We’ll scrape fields highlighted in blue in this example
Idealista is a pure HTML website with a very convenient styling markup which we can take advantage in our scraper. For example, if we right-click on the price and inspect the HTML element we can see how clear the HTML structure is:
illustration of idealista’s source page
We can see that all of the data points are under clear class names like info-data-price for price or main-info__title-main for property name.
Parsing HTML with CSS Selectors
Introduction to using CSS selectors to parse web-scraped content. Best practices, available tools and common challenges by interactive examples.
Let’s scrape it:
import asyncio
import json
import re
from typing import Dict, List
from collections import defaultdict
from urllib.parse import urljoin
import httpx
from parsel import Selector
from typing_extensions import TypedDict
# Establish persisten HTTPX session with browser-like headers to avoid blocking
BASE_HEADERS = {
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"accept-language": "en-US;en;q=0.9",
"accept-encoding": "gzip, deflate, br",
}
session = httpx.AsyncClient(headers=BASE_HEADERS, follow_redirects=True)
# type hints fo expected results so we can visualize our scraper easier:
class PropertyResult(TypedDict):
url: str
title: str
location: str
price: int
currency: str
description: str
updated: str
features: Dict[str, List[str]]
images: Dict[str, List[str]]
plans: List[str]
def parse_property(response: httpx.Response) -> PropertyResult:
"""parse Idealista.com property page"""
# load response's HTML tree for parsing:
selector = Selector(text=response.text)
css = lambda x: selector.css(x).get("").strip()
css_all = lambda x: selector.css(x).getall()
data = {}
# Meta data
data["url"] = str(response.url)
# Basic information
data['title'] = css("h1 .main-info__title-main::text")
data['location'] = css(".main-info__title-minor::text")
data['currency'] = css(".info-data-price::text")
data['price'] = int(css(".info-data-price span::text").replace(",", ""))
data['description'] = "\n".join(css_all("div.comment ::text")).strip()
data["updated"] = selector.xpath(
"//p[@class='stats-text']"
"[contains(text(),'updated on')]/text()"
).get("").split(" on ")[-1]
# Features
data["features"] = {}
# first we extract each feature block like "Basic Features" or "Amenities"
for feature_block in selector.css(".details-property-h2"):
# then for each block we extract all bullet points underneath them
label = feature_block.xpath("text()").get()
features = feature_block.xpath("following-sibling::div[1]//li")
data["features"][label] = [
''.join(feat.xpath(".//text()").getall()).strip()
for feat in features
]
# Images
# the images are tucked away in a javascript variable.
# We can use regular expressions to find the variable and parse it as a dictionary:
image_data = re.findall(r"fullScreenGalleryPics\s*:\s*(\[.+?\]),",
response.scrape_result['content']
)[0]
# we also need to replace unquoted keys to quoted keys (i.e. title -> "title"):
images = json.loads(re.sub(r'(\w+?):([^/])', r'"\1":\2', image_data))
data['images'] = defaultdict(list)
data['plans'] = []
for image in images:
url = urljoin(str(response.url), image['imageUrl'])
if image['isPlan']:
data['plans'].append(url)
else:
data['images'][image['tag']].append(url)
return data
async def scrape_properties(urls: List[str]) -> List[PropertyResult]:
"""Scrape Idealista.com properties"""
properties = []
to_scrape = [session.get(url) for url in urls]
# tip: asyncio.as_completed allows concurrent scraping - super fast!
for response in asyncio.as_completed(to_scrape):
response = await response
print(response.status_code)
if response.status_code != 200:
print(f"can't scrape property: {response.url}")
continue
properties.append(parse_property(response))
return propertiesIf you are experiencing errors while running the Python code tabs, it may be due to getting blocked. To bypass blocking, use the ScrapFly code tabs instead.
In this demonstration, we used several CSS and XPath selectors with parsel to extract property details like price, description, features and more.
However, the images are where things get more complex. For image carousels, many websites use JavaScript to generate dynamic HTML on demand. Idealista follows this pattern and hides all image URLs in a JavaScript variable, then displays them using JavaScript.
To scrape this, we utilized a regular expression pattern to find the hidden JavaScript variable, then loaded it as a Python dictionary object and parsed the image and floor plans.The visible HTML doesn’t always represent the whole dataset available on the page. In this article, we’ll be taking a look at scraping of hidden web data. What is it and how can we scrape it using Python?
For the scraping implementation, we leveraged asynchronous capabilities of httpx and asyncio.as_completed to schedule multiple properties concurrently, making our scraper incredibly fast!
Next, let’s take a look at how we can scale up this scraper by implementing exploration functionality.
Finding Idealista Properties
There are several approaches to discovering properties listed on Idealista. The most reliable and popular method is to explore by geographical area. In this section, we’ll examine how to scrape property listings with a bit of crawling – we’ll explore the location directory.
To find the location directory we can scroll to the bottom of the page:
screenshot of idealista location directory page
Location directory found at the bottom of the page.
Each link leads to a province listing URL which further leads to area listings URLs. We can easily scrape this with the same CSS selector technique we’ve used previously:
def parse_province(response: httpx.Response) -> List[str]:
"""parse province page for area search urls"""
selector = Selector(text=response.text)
urls = selector.css("#location_list li>a::attr(href)").getall()
return [urljoin(str(response.url), url) for url in urls]
async def scrape_provinces(urls: List[str]) -> List[str]:
"""
Scrape province pages like:
https://www.idealista.com/en/venta-viviendas/malaga-provincia/con-chalets/municipios
for search page urls like:
https://www.idealista.com/en/venta-viviendas/marbella-malaga/con-chalets/
"""
to_scrape = [session.get(url) for url in urls]
search_urls = []
async for response in asyncio.as_completed(to_scrape):
search_urls.extend(parse_province(await response))
return search_urlsThis scraper will process all area pages for given provinces. To discover all property listings, we’d simply need to scrape all provinces. Next, let’s scrape the search results page itself:
def parse_search(response: httpx.Response) -> List[str]:
"""Parse search result page for 30 listings"""
selector = Selector(text=response.text)
urls = selector.css("article.item .item-link::attr(href)").getall()
return [urljoin(str(response.url), url) for url in urls]
async def scrape_search(url: str, paginate=True, max_pages: int = None) -> List[str]:
"""
Scrape search urls like:
https://www.idealista.com/en/venta-viviendas/marbella-malaga/con-chalets/
for proprety urls
"""
first_page = await session.get(url)
property_urls = parse_search(first_page)
if not paginate:
return property_urls
total_results = first_page.selector.css("h1#h1-container").re(": (.+) houses")[0]
total_pages = math.ceil(int(total_results.replace(",", "")) / 30)
if total_pages > 60:
print(f"search contains more than max page limit ({total_pages}/60)")
total_pages = 60
# scrape all available pages in the search if max_scrape_pages is None or max_scrape_pages > total_pages
if max_pages and max_pages < total_pages:
total_pages = max_pages
else:
total_pages = total_pages
print(f"scraping {total_pages} of search results concurrently")
to_scrape = [
session.get(first_page.url + f"pagina-{page}.htm")
for page in range(2, total_pages + 1)
]
async for response in asyncio.as_completed(to_scrape):
property_urls.extend(parse_search(await response))
return property_urlsFor scraping paginated content like the area results pages, we first scrape the initial page to extract the total result count. Then, we can scrape the remaining pages concurrently, retrieving all listings in just a few seconds!
With this discovery scraper combined with our previous property scraper, we can collect all of the existing real estate data on Idealista.com – though what if we want to be the first to know about new property listings? Next, let’s examine how we can scrape Idealista’s search results.
How to Scrape Idealista Search
In this section, we’ll scrape Idealista’s search pages. These search pages enable finding specific property listings and sorting them. For example, let’s find properties in Malaga, Spain:
screenshot of idealista search pages
To build an Idealista scraper for search pages, we’ll request search pages and parse their results while incrementing the pagina parameter for pagination:
import json
import math
import httpx
import asyncio
from typing import Dict, List
# Establish persisten HTTPX session with browser-like headers to avoid blocking
BASE_HEADERS = {
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"accept-language": "en-US;en;q=0.9",
"accept-encoding": "gzip, deflate, br",
}
session = httpx.AsyncClient(headers=BASE_HEADERS, follow_redirects=True)
def parse_search_data(response) -> List[Dict]:
"""parse search result data"""
selector = Selector(response.text)
total_results = selector.css("h1#h1-container").re(": (.+) houses")[0]
max_pages = math.ceil(int(total_results.replace(",", "")) / 30)
max_pages = 60 if max_pages > 60 else max_pages
search_data = []
for box in selector.xpath("//section[contains(@class, 'items-list')]/article[contains(@class, 'item')]"):
ad = box.xpath(".//p[@class='adv_txt']") # ignore ad listings
if ad:
continue
price = box.xpath(".//span[contains(@class, 'item-price')]/text()").get()
parking = box.xpath(".//span[@class='item-parking']").get()
company_url = box.xpath(".//picture[@class='logo-branding']/a/@href").get()
search_data.append({
"title": box.xpath(".//div/a/@title").get(),
"link": "https://www.idealista.com" + box.xpath(".//div/a/@href").get(),
"picture": box.xpath(".//img/@src").get(),
"price": int(price.replace(",", '')) if price else None,
"currency": box.xpath(".//span[contains(@class, 'item-price')]/span/text()").get(),
"parking_included": True if parking else False,
"details": box.xpath(".//div[@class='item-detail-char']/span/text()").getall(),
"description": box.xpath(".//div[contains(@class, 'item-description')]/p/text()").get().replace('\n', ''),
"tags": box.xpath(".//div[@class='listing-tags-container']/span/text()").getall(),
"listing_company": box.xpath(".//picture[@class='logo-branding']/a/@title").get(),
"listing_company_url": "https://www.idealista.com" + company_url if company_url else None
})
return {"max_pages": max_pages, "search_data": search_data}
async def scrape_search(url: str, max_scrape_pages: int = None) -> List[Dict]:
"""scrape Idealista search results"""
first_page = await session.get(url)
assert first_page == 200, "request is blocked, use ScrapFly code tab"
data = parse_search_data(first_page)
search_data = data["search_data"]
max_pages = data["max_pages"]
# get the number of total pages to scrape
if max_scrape_pages and max_scrape_pages < max_pages:
max_pages = max_scrape_pages
# scrape the remaining pages concurrently
to_scrape = [
session(url + f"pagina-{page}.htm")
for page in range(2, max_pages + 1)
]
print(f"scraping search pagination, {max_pages - 1} pages remaining")
for response in asyncio.as_completed(to_scrape):
search_data.extend(parse_search_data(await response)["search_data"])
print(f"scraped {len(search_data)} property listings from search pages")
return search_dataAbove, we define a parse_search_data utility function. It parses the HTML page using XPath selectors to extract the search results. We also use the scrape_search function to paginate search pages by requesting the first page, adding the remaining pages to a scraping list, and then scraping them concurrently.
Here’s an example results from the above Idealista scraper:
We scraped Idealista data from discovery, property, and search pages – all that’s left is to scale our scraper. If we were to increase our scraping load, Idealista would likely block us, so let’s look at how to avoid blocking using Webparsers web scraping API next.
Bypass Idealista Blocking with Webparsers
As we’ve seen, scraping Idealista.com using Python is quite straightforward, though when scraping at scale, our scrapers are likely to be blocked or asked to solve captchas.
scrapfly middleware
ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.
- Anti-bot protection bypass – scrape web pages without blocking!
- Rotating residential proxies – prevent IP address and geographic blocks.
- JavaScript rendering – scrape dynamic web pages through cloud browsers.
- Full browser automation – control browsers to scroll, input and click on objects.
- Format conversion – scrape as HTML, JSON, Text, or Markdown.
- Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.
For example, we can use the scrapfly-sdk python package and the Anti Scraping Protection Bypass feature. First, let’s install scrapfly-sdk using pip:
$ pip install scrapfly-sdkTo take advantage of ScrapFly’s API in our Idealista web scraper all we need to do is change our httpx session code with scrapfly-sdk client requests:
import httpx
response = httpx.get("some idealista.com URL")
# in ScrapFly SDK becomes
from scrapfly import ScrapflyClient, ScrapeConfig
client = ScrapflyClient("YOUR SCRAPFLY KEY")
result = client.scrape(ScrapeConfig(
"some Idealista.ocm url",
# we can select specific proxy country like Spain:
country="ES",
# and enable anti scraping protection bypass:
asp=True
))For more on how to scrape Idealista.com using ScrapFly, see the Full Scraper Code section.
FAQ
To wrap this guide up, let’s take a look at some frequently asked questions about web scraping Idealista.com data:
Is it legal to scrape Idealista.com?
Yes. Idealista.com’s data is publicly available; we’re not extracting anything personal or private. Scraping Idealista.com at slow, respectful rates is perfectly legal and ethical.
That being said, attention should be paid to GDRP compliance in the EU when scraping personal data like (seller’s name, phone number etc). For more, see our Is Web Scraping Legal? article.
Does Idealista.com have a public API?
No, Idealista.com (and its sister websites) do not offer a public API for property data. However, as demonstrated in this guide, it’s straightforward to scrape and crawl using a little bit of Python.
Idealista Scraping Summary
In this web scraping tutorial, we built a comprehensive Idealista scraper for real estate property data. We started by scraping a single property page and parsing details using CSS and XPath selectors.
Then, we explored how to find properties using Idealista’s directory and search system. We developed a small web crawler that can crawl and scrape all property listings in provided provinces of Spain.
Finally, we examined how to track new listings being posted on Idealista by creating a looping scraper that constantly checks for new listings.
For all of this, we used Python with httpx and parsel packages and to avoid being blocked we used Webparsers API that intelligently configures every web scraper connection to avoid blocking.
For more about ScrapFly, see our documentation and try it out for FREE!