Skip to main content

Webparsers.com

Webparsers Homepage
Login
Menu
Menu
Login

How to Scrape eBay Using Python (2026 Update)

15 min read

Table of Contents

Ebay stands as the world’s largest peer-to-peer e-commerce marketplace, making it a compelling target for public data collection!

This comprehensive guide will walk you through scraping Ebay search and product listing pages to extract various details including pricing information, variant data, product features, and descriptions.

We’ll leverage Python alongside several community packages and implement sophisticated parsing techniques. Let’s dive in!

Key Takeaways

Master ebay scraper development using Python with httpx and parsel, extracting product data from hidden web data and handling anti-bot measures for comprehensive e-commerce data collection.

  • Reverse engineer eBay’s search API endpoints by intercepting browser network requests and analyzing JSON response structures
  • Parse dynamic JSON data embedded in HTML using XPath selectors for product details and variants
  • Bypass eBay’s anti-scraping measures with realistic headers, user agents, and request spacing
  • Extract structured product data including titles, prices, descriptions, and seller information
  • Implement exponential backoff retry logic with 403 status code detection for rate limiting
  • Handle multi-variant products and dynamic pricing through advanced JSON parsing and data extraction techniques

Why Scrape Ebay?

Ebay represents one of the world’s largest product marketplaces, particularly for niche and rare items. This positioning makes Ebay an excellent target for e-commerce data analytics.

Scraping Ebay data enables various use cases, including:

  • Competitor analysis by gathering data on competitors’ sales and reviews.
  • Market research by tracking product prices for hot deals or trends.
  • Empowered navigation through automated search patterns and custom alerts.

For further details, refer to our introduction on web scraping use cases.

Setup

Web scraping Ebay requires utilizing several Python community packages:

In this tutorial, we’ll be working with Python alongside two essential community libraries:

  • webparsers-sdk: A Python SDK for WebParsers, a web scraping API that bypasses web scraping blocking
  • jmespath: For refining and parsing JSON datasets
  • nested-lookup: To find nested keys in the Ebay JSON datasets

The above packages can be installed using the below pip command:

$ pip install webparsers-sdk jmespath nested-lookup

Scraping Ebay Listings

Let’s begin by scraping Ebay for individual listing pages. Ebay listings come in two main types:

  • Single variant listings with fixed selections
  • Multiple variant listings with different selections, like tech devices

First, we’ll focus on scraping single variants since they present a more straightforward extraction process.

We’ll be utilizing single variants since they offer more straightforward extraction. Let’s examine this product as an example, where we’ll extract data from the following fields:

We’ll capture the most essential fields: pricing, description and product and seller details

In the image above we marked our target fields and to construct CSS selectors for these fields we can utilize the Browser Developer Tools (F12 key or right click → inspect option).

To scrape the above Ebay listing data, we’ll employ CSS and XPath selectors:

import json
import os
import re
import asyncio

from typing import Dict, List
from nested_lookup import nested_lookup
from webparsers import ScrapeApiResponse, ScrapeConfig, WebparsersClient, WebparsersScrapeError

BASE_CONFIG = {
    "asp": True,
    "country": "US",
    "lang": ["en-US"]
}

WEBPARSERS = WebparsersClient(key=os.environ["API_KEY"])

def parse_product(result: ScrapeApiResponse):
    """Parse Ebay's product listing page for core product data"""
    sel = result.selector
    css_join = lambda css: "".join(sel.css(css).getall()).strip()  # join all selected elements
    css = lambda css: sel.css(css).get("").strip()  # take first selected element and strip of leading/trailing spaces

    item = {}
    item["url"] = css('link[rel="canonical"]::attr(href)')
    item["id"] = item["url"].split("/itm/")[1].split("?")[0]  # we can take ID from the URL
    item["price_original"] = css(".x-price-primary>span::text")
    item["price_converted"] = css(".x-price-approx__price ::text")  # ebay automatically converts price for some regions

    item["name"] = css_join("h1 span::text")
    item["seller_name"] = sel.xpath("//div[contains(@class,'info__about-seller')]/a/span/text()").get()
    item["seller_url"] = sel.xpath("//div[contains(@class,'info__about-seller')]/a/@href").get().split("?")[0]
    item["photos"] = sel.css('.ux-image-filmstrip-carousel-item.image img::attr("src")').getall()  # carousel images
    item["photos"].extend(sel.css('.ux-image-carousel-item.image img::attr("src")').getall())  # main image
    # description is an iframe (independant page). We can keep it as an URL or scrape it later.
    item["description_url"] = css("iframe#desc_ifr::attr(src)")
    # feature details from the description table:
    feature_table = sel.css("div.ux-layout-section--features")
    features = {}
    for feature in feature_table.css("dl.ux-labels-values"):
        # iterate through each label of the table and select first sibling for value:
        label = "".join(feature.css(".ux-labels-values__labels-content > div > span::text").getall()).strip(":\n ")
        value = "".join(feature.css(".ux-labels-values__values-content > div > span *::text").getall()).strip(":\n ")
        features[label] = value
    item["features"] = features
    return item


async def scrape_product(url: str) -> Dict:
    """Scrape ebay.com product listing page for product data"""
    print(f"scraping product: {url}")
    page = await WEBPARSERS.async_scrape(ScrapeConfig(url, **BASE_CONFIG))
    product = parse_product(page)
    return product


async def main():
    product_data = await scrape_product("https://www.ebay.com/itm/332562282948")

    # save the results to a json file
    with open("product_data.json", "w", encoding="utf-8") as f:
        json.dump(product_data, f, indent=2, ensure_ascii=False)


if __name__ == "__main__":
    asyncio.run(main())

Let’s break down the above Ebay scraping code. We begin by defining a new Webparsers client and define two functions:

  • parse_product: to parse the product HTML pages using CSS and XPath selectors
  • scrape_product: To request Ebay product pages using Webparsers to bypass its antibot and retrieve the HTML

Below is example output of the Ebay data retrieved

Next, for products with variants we’ll need to go a bit further and extract the page’s hidden web data. It might seem like a complex process, though we’ll cover it step-by-step!

Scraping Ebay Listing Variant Data

Ebay’s listings can contain multiple products through a feature called variants. For example, let’s examine this iPhone listing:

Listings with variants have multiple selection options

We can observe several variant options: model, storage capacity, and color. These options get updated using JavaScript each time we select one.

Ebay utilizes JavaScript to update the page with different pricing every time we choose a different option. This means that the variant data exists in a JavaScript variable. Extracting these data is commonly referred to as hidden web data.

We’ll briefly mention the hidden web data extraction in this guide. For the complete details, refer to our dedicated tutorial.

How to Scrape Hidden Web Data

The visible HTML doesn’t always represent the complete dataset available on the page. In this article, we’ll be examining the scraping of hidden web data. What is it and how can we scrape it using Python?

To scrape the product variant data, we’ll extract them as JSON under hidden script tags:

import json
import os
import re
import asyncio

from typing import Dict, List
from collections import defaultdict
from nested_lookup import nested_lookup
from webparsers import ScrapeApiResponse, ScrapeConfig, WebparsersClient

BASE_CONFIG = {
    "asp": True,
    "country": "US",
    "lang": ["en-US"]
}

WEBPARSERS = WebparsersClient(key=os.environ["API_KEY"])

def _find_json_objects(text: str, decoder=json.JSONDecoder()):
    """Find JSON objects in text, and generate decoded JSON data"""
    pos = 0
    while True:
        match = text.find("{", pos)
        if match == -1:
            break
        try:
            result, index = decoder.raw_decode(text[match:])
            yield result
            pos = match + index
        except ValueError:
            pos = match + 1

def parse_variants(result: ScrapeApiResponse) -> dict:
    """
    Parse variant data from Ebay's listing page of a product with variants.
    This data is located in a js variable MSKU hidden in a