How to Web Scrape Leboncoin.fr using Python

Leboncoin.fr stands as one of France’s largest marketplace platforms for peer-to-peer commerce. As a significant data source, it presents considerable scraping challenges due to sophisticated anti-bot protection mechanisms.

This guide demonstrates how to extract data from leboncoin.fr effectively while circumventing detection systems. We’ll cover scraping techniques for both search results and individual listing pages. Let’s get started!

Key Takeaways

Develop robust leboncoin scrapers using Python to collect marketplace information, real estate listings, and search data while overcoming anti-scraping defenses for thorough French market intelligence.

• Analyze Leboncoin’s API structure through browser network inspection and JSON response examination
• Parse structured marketplace information including pricing, geographic data, and product specifications from French listings
• Handle pagination workflows and search parameter configuration for complete marketplace data extraction
• Set up proxy rotation and browser fingerprint management to prevent detection and throttling
• Leverage specialized platforms like Webparsers for automated Leboncoin scraping with blocking prevention
• Build data validation and error management systems for dependable French marketplace data collection

Why Extract Data from Leboncoin.fr?

Leboncoin.fr hosts millions of advertisements across diverse categories, spanning household items and vehicles to property listings. Web scraping this platform offers valuable opportunities for:

Market Intelligence

Listing information can be extracted and examined to understand market trends, pricing behaviors, and consumer demand patterns.

Competitive Analysis

Extracting Leboncoin data enables businesses to gain strategic advantages through analysis of competitor offerings and positioning.

Price Monitoring

Buyers and sellers can utilize web scraping to monitor product pricing evolution, enabling prediction of future price movements and identification of favorable deals.

Inventory Management

Merchants can extract Leboncoin data to synchronize their inventory systems with products available on the platform.

Project Configuration

For this leboncoin extraction tutorial, we’ll utilize several Python libraries:

• Webparsers-sdk – a web scraping API and Python SDK that enables large-scale scraping without blocking
• parsel – an HTML parsing library

We’ll execute our scrapers asynchronously using Python’s asyncio, which significantly enhances web scraping performance.

Install these libraries using pip:

pip install webparsers-sdk parsel

Overcome Leboncoin Scraping Protection With Webparsers

Leboncoin.fr implements advanced protection systems that identify web scrapers. For instance, attempting to scrape leboncoin using basic headless browser automation with the Playwright library for Python:

from playwright.sync_api import sync_playwright

with sync_playwright() as playwight:
    # Lanuch a chrome browser
    browser = playwight.chromium.launch(headless=False)
    page = browser.new_page()
    # Go to leboncoin.fr
    page.goto("https://www.leboncoin.fr")
    # Take a screenshot
    page.screenshot(path="screenshot.png")

The platform identified us as automated scrapers and presented a captcha verification challenge:

Leboncoin.fr scraping block page – captcha challenge required

To circumvent leboncoin.fr web scraping protection, consider using advanced scraping solutions!

Webparsers offers web scraping, screenshot, and extraction APIs for large-scale data collection.

• Anti-bot protection bypass – scrape web pages without blocking!
• Rotating residential proxies – prevent IP address and geographic blocks.
• JavaScript rendering – scrape dynamic web pages through cloud browsers.
• Full browser automation – control browsers to scroll, input and click on objects.
• Format conversion – scrape as HTML, JSON, Text, or Markdown.
• Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

For example, using the Webparsers asp feature with the Webparsers SDK allows easy bypass of leboncoin.fr scraper blocking:

from webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse

webparsers = WebparsersClient(key="Your API key")

api_response: ScrapeApiResponse = webparsers.scrape(
    ScrapeConfig(
        url="https://www.leboncoin.fr",
        # Cloud headless browser similar to Playwright
        render_js=True,
        # Bypass anti scraping protection
        asp=True,
        # Set the geographical location to France
        country="FR",
    )
)
# Print the website's status code
print(api_response.upstream_status_code)
"200"

Now that we can bypass leboncoin.fr blocking with Webparsers, let’s create a comprehensive leboncoin scraper.

How to Extract Leboncoin Search Results?

Let’s examine how the search functionality operates on leboncoin.fr.

When navigating to the homepage and performing a keyword search, we encounter a results page like this:

example Leboncoin search page

This sample search page for real estate listings implements pagination using the following URL pattern:

https://www.leboncoin.fr/recherche?text=maison&page=1

We’ll utilize this URL structure as our primary search endpoint and leverage the page parameter to iterate through search result pages.

Let’s start by examining how to extract data from the initial page, then implement pagination to scrape subsequent results.

For result extraction, we’ll employ a hidden web data methodology. Rather than using parsing selectors like XPath or CSS selectors, we’ll retrieve all data in JSON format directly from script tags embedded in the HTML.

To find this script tag, open browser developer tools by pressing F12. Then scroll through the page until you locate the script tag with the __NEXT_DATA__ ID:

ad data in script tag

We’ll target this script tag from the HTML and extract its data within our scraper:

from Webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse
import asyncio
from typing import Dict, List
import json

SCRAPER = WebparsersClient(key="Your API key")

# Webparsers config
BASE_CONFIG = {
    # bypass web scraping blocking
    "asp": True,
    # set the proxy location to France
    "country": "fr",
}

def parse_search(result: ScrapeApiResponse):
    """parse search result data from nextjs cache"""
    # select the __NEXT_DATA__ script from the HTML
    next_data = result.selector.css("script[id='__NEXT_DATA__']::text").get()
    # extract ads listing data from the search page
    ads_data = json.loads(next_data)["props"]["pageProps"]["initialProps"]["searchData"]["ads"]
    return ads_data

async def scrape_search(url: str) -> List[Dict]:
    """scrape leboncoin search"""
    print(f"scraping search {url}")
    first_page = await SCRAPER.async_scrape(ScrapeConfig(url, **BASE_CONFIG))
    search_data = parse_search(first_page)
    # print the data in JSON format
    print(json.dumps(search_data, indent=2))

# run the scraping search function
asyncio.run(scrape_search(url="https://www.leboncoin.fr/recherche?text=coffe"))

Here, we utilize the parse_search function to extract and process search data from the HTML. Next, we employ scrape_search to extract data from the initial search page using Webparsers. Finally, we output the results in JSON format and execute the code using asyncio.

The current leboncoin scraper extracts data from only the first search page. Let’s enhance it to scrape multiple pages:

from Webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse
from typing import Dict, List
import asyncio
import json

Webparsers = WebparsersClient(key="Your API key")

# Webparsers config
BASE_CONFIG = {
    # bypass web scraping blocking
    "asp": True,
    # set the proxy location to France
    "country": "fr",
}

def parse_search(result: ScrapeApiResponse):
    """parse search result data from nextjs cache"""
    # select the __NEXT_DATA__ script from the HTML
    next_data = result.selector.css("script[id='__NEXT_DATA__']::text").get()
    # extract ads listing data from the search page
    ads_data = json.loads(next_data)["props"]["pageProps"]["initialProps"]["searchData"]["ads"]
    return ads_data

async def scrape_search(url: str, max_pages: int) -> List[Dict]:
    """scrape leboncoin search"""
    print(f"scraping search {url}")
    first_page = await Webparsers.async_scrape(ScrapeConfig(url, **BASE_CONFIG))
    search_data = parse_search(first_page)
    # add the ramaining pages in a scraping list
    _other_pages = [
        ScrapeConfig(f"{first_page.context['url']}&page={page}", **BASE_CONFIG)
        for page in range(2, max_pages + 1)        
    ]
    # scrape the remaining pages concurrently
    async for result in Webparsers.concurrent_scrape(_other_pages):
        ads_data = parse_search(result)
        search_data.extend(ads_data)    
    print(json.dumps(search_data, indent=2))

# run the scraping search function
asyncio.run(scrape_search(url="https://www.leboncoin.fr/recherche?text=coffe", max_pages=2))

Here, we introduce a max_pages parameter to the scrape_search function, which determines the number of search pages to process. The extraction result is a comprehensive list containing all advertisement data discovered across two search pages:

Output

We successfully extracted all listing data using Leboncoin’s search functionality. Next, let’s examine how to scrape individual listing pages!

How to Extract Leboncoin.fr Listing Advertisements?

While listing data on search pages and individual listing pages contains identical information, the location within the HTML structure differs. We’ll need to modify the object keys used to access the hidden web data:

from Webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse
from typing import Dict
import asyncio
import json

Webparsers = WebparsersClient(key="Your API key")

# Webparsers config
BASE_CONFIG = {
    # bypass web scraping blocking
    "asp": True,
    # set the proxy location to France
    "country": "fr",
}


def parse_ad(result: ScrapeApiResponse):
    """parse ad data from nextjs cache"""
    next_data = result.selector.css("script[id='__NEXT_DATA__']::text").get()
    # extract ad data from the ad page
    ad_data = json.loads(next_data)["props"]["pageProps"]["ad"]
    return ad_data


async def scrape_ad(url: str, _retries: int = 0) -> Dict:
    """scrape ad page"""
    print(f"scraping ad {url}")
    try:
        result = await Webparsers.async_scrape(ScrapeConfig(url, **BASE_CONFIG))
        ad_data = parse_ad(result)
    except:
        print("retrying failed request")
        if _retries < 2:
            return await scrape_ad(url, _retries=_retries + 1)
    return ad_data

Run the code

Similar to our previous approach, we employ the parse_ad function to extract and process advertisement data from the listing page HTML. Then, we use the scrape_ad function to extract the ad page data using Webparsers. Here is the output we obtained:

Output

Excellent – we can successfully extract leboncoin.fr data from both search results and individual advertisement pages!

FAQ

To conclude this guide, let’s address some common questions about leboncoin.fr web scraping.

Is it legal to scrape leboncoin.fr?

Yes, all advertisement data on leboncoin is publicly accessible, making it legal to scrape as long as you maintain reasonable scraping rates. However, you should consider GDPR compliance in the EU when extracting personal information, such as seller details. For more details, refer to our article on web scraping legality.

Is there a public API for leboncoin.fr?

Currently, leboncoin.fr does not provide a publicly available API. However, scraping leboncoin.fr is relatively straightforward and can be used to develop your own web scraping API solution.

How to prevent leboncoin.fr web scraping detection?

Multiple factors contribute to web scraping detection including headers, IP addresses and security handshakes. To avoid leboncoin.fr web scraping blocking, attention to these technical details is essential. For more information, refer to our previous guide on scraping without getting blocked.

Leboncoin.fr Scraping Summary

Leboncoin.fr represents one of France’s most prominent marketplace platforms for advertisements. It implements sophisticated protection systems that detect and block web scrapers, necessitating the use of advanced anti-scraping solutions.

In this article, we explored comprehensive techniques for scraping leboncoin.fr to extract advertisement and search data. We also demonstrated how to circumvent leboncoin web scraping protection using Webparsers.

Legal Disclaimer and Precautions

This tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect and here’s a good summary of what not to do:

• Do not scrape at rates that could damage the website.
• Do not scrape data that’s not available publicly.
• Do not store PII of EU citizens who are protected by GDPR.
• Do not repurpose the entire public datasets which can be illegal in some countries.

Webparsers does not offer legal advice but these are good general rules to follow in web scraping and for more you should consult a lawyer.