How to Scrape Etsy.com Product, Shop and Search Data

Etsy.com is a global online marketplace where users can buy and sell handmade and vintage items. It’s a valuable data target though it can be challenging to scrape due to its high level of protection.

In this comprehensive guide on web scraping Etsy, we’ll extract items and review data from product, shop and search pages. Additionally, we’ll explore methods to circumvent Etsy scraping blocks. Let’s dive in!

Key Takeaways

Master etsy scraper development with Python to extract product data, reviews, and shop information while bypassing anti-scraping measures for comprehensive e-commerce analysis.

• Reverse engineer Etsy’s hidden JSON data embedded in script tags for product details and variants
• Extract structured review data including ratings, comments, and reviewer information from product pages
• Handle Etsy’s anti-scraping measures with realistic headers, user agents, and request spacing
• Parse dynamic search results and pagination to collect comprehensive product listings
• Implement error handling and retry logic for rate limiting and temporary blocking scenarios
• Use Webparsers SDK for automated Etsy scraping with built-in anti-blocking protection

Why Scrape Etsy.com?

If you are a buyer looking to purchase specific items, manually browsing thousands of product listings to identify the best deal can be tedious and time-consuming. With etsy.com scraping, we can retrieve thousands of listings and compare them efficiently, enabling better decision-making.

Web scraping etsy.com also enables businesses and sellers to understand and analyze market trends to gain insights into consumer behavior patterns.

Furthermore, scraping sellers’ and shops’ data from Etsy allows business owners to analyze their competitors’ and market peers’ items, inventory and pricing strategies. This leads to making strategic decisions and gaining a competitive advantage.

Project Setup

To scrape etsy.com, we’ll use a few Python libraries.

• webparsers-sdk for bypassing etsy.com web scraping blocking using Webparsers’ web scraping API.
• asyncio for running our code in an asynchronous fashion, increasing our web scraping speed.

As asyncio comes included with Python, so we only have to install webparsers-sdk using the following pip command:

pip install webparsers-sdk

How to Scrape Etsy Listings

Let’s begin by scraping Etsy.com listing pages. Navigate to any listing page on the website and you will encounter a page similar to this:

Item listing page on Etsy.com

To scrape listing pages’ data, we’ll extract all the information directly in JSON format rather than parsing each data point from the HTML.

To view the hidden listing data, open the browser developer tools (by pressing the F12 key) to view the page HTML. Then, scroll down until you find the script tag with the application/ld+json type. The data inside this tag looks like this:

Hidden data on product listing page

This data is identical to what appears on the web page but before getting rendered into the HTML, commonly known as hidden web data. To scrape etsy.com listing pages, we’ll select this script tag and extract the internal data as JSON directly:

import os
import math
import json
import asyncio

from typing import Dict, List
from webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse

BASE_CONFIG = {
    "asp": True,
    "country": "US",
}

Scraper = WebparsersClient(key=os.environ["API_KEY"])

def parse_product_page(response: ScrapeApiResponse) -> Dict:
    """parse hidden product data from product pages"""
    selector = response.selector
    script = selector.xpath("//script[contains(text(),'offers')]/text()").get()
    if not script:
        print(f"Could not find product data script on {response.context['url']}")
        return {}
    data = json.loads(script)
    return data


async def scrape_product(urls: List[str]) -> List[Dict]:
    """scrape trustpilot company pages"""
    products = []
    # add the product page URLs to a scraping list
    to_scrape = [ScrapeConfig(url, **BASE_CONFIG) for url in urls]
    # scrape all the product pages concurrently
    async for response in Scraper.concurrent_scrape(to_scrape):
        data = parse_product_page(response)
        products.append(data)
    print(f"scraped {len(products)} product listings from product pages")
    return products


async def main():
    product_data = await scrape_product(
        urls=[
            "https://www.etsy.com/listing/1552627931",
            "https://www.etsy.com/listing/529765307",
            "https://www.etsy.com/listing/949905096",
        ]
    )

    # save the results to a json file
    with open("product_data.json", "w", encoding="utf-8") as file:
        json.dump(product_data, file, indent=2, ensure_ascii=False)


if __name__ == "__main__":
    asyncio.run(main())

Here, we initialize an httpx client with basic browser headers and define two functions:

• parse_product_page() for parsing the HTML and selecting the JSON data inside the script tag.
• scrape_product() for scraping the product pages by adding the page URLs to a scraping list and scraping them concurrently.

Here is a sample output of the result we obtained:

Example output

Excellent! Our Etsy scraper retrieved all the product data and several reviews with just a few lines of code. Next, we’ll scrape shop data!

How to Scrape Etsy Shops

Shop pages on etsy.com contain data about the products sold by a shop alongside the shop reviews. Similar to product listing pages, shop page data are also found under script tags:

Hidden data on shop page

Just like in the previous section, we’ll scrape etsy.com shop pages by extracting the data directly from the above script tag:

import os
import math
import json
import asyncio

from typing import Dict, List
from webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse

BASE_CONFIG = {
    "asp": True,
    "country": "US",
}

Scrapers = WebparsersClient(key=os.environ["API_KEY"])

def parse_shop_page(response: ScrapeApiResponse) -> Dict:
    """parse hidden shop data from shop pages"""
    selector = response.selector
    script = selector.xpath("//script[contains(text(),'itemListElement')]/text()").get()
    data = json.loads(script)
    return data


async def scrape_shop(urls: List[str]) -> List[Dict]:
    shops = []
    # add the shop page URLs to a scraping list
    to_scrape = [ScrapeConfig(url, **BASE_CONFIG) for url in urls]
    # scrape all the shop pages concurrently
    async for response in Scraper.concurrent_scrape(to_scrape):
        data = parse_shop_page(response)
        data["url"] = response.context["url"]
        shops.append(data)
    print(f"scraped {len(shops)} shops from shop pages")
    return shops


async def main():
    shop_data = await scrape_shop(
        urls=[
            "https://www.etsy.com/shop/FalkelDesign",
            "https://www.etsy.com/shop/JoshuaHouseCrafts",
            "https://www.etsy.com/shop/Oakywood",
        ]
    )

    # save the results to a json file
    with open("shop_data.json", "w", encoding="utf-8") as file:
        json.dump(shop_data, file, indent=2, ensure_ascii=False)


if __name__ == "__main__":
    asyncio.run(main())

The above code follows the same structure as the Etsy scraper we wrote earlier. We have only modified the naming conventions and the XPath selector.

Here is a sample output of the result we obtained:

Sample output

Great! We successfully scraped product and shop pages from etsy.com. The final component of our Etsy scraper is the search pages. Let’s explore that!

How to Scrape Etsy Search

In this section, we’ll scrape item listing data from search pages. But first, let’s examine what the search pages on etsy.com look like. Search for any product on the website and you will encounter a page similar to this:

Search page on Etsy.com

Unlike the product and shop pages, hidden data on search pages don’t provide all the search results. For example, here is hidden data on a search page. It contains 8 product listings, but the actual page displays 64 product listings:

So, to scrape Etsy.com search pages, we need to parse each listing data from the HTML. Let’s proceed with that approach.

import os
import math
import json
import asyncio

from typing import Dict, List
from webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse

BASE_CONFIG = {
    "asp": True,
    "country": "US",
}

Scraper = WebparsersClient(key=os.environ["API_KEY"])

def strip_text(text):
    """remove extra spaces while handling None values"""
    if text != None:
        text = text.strip()
    return text


def parse_search(response: ScrapeApiResponse) -> Dict:
    """parse data from Etsy search pages"""
    selector = response.selector
    data = []
    script = json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())
    # get the total number of pages
    total_listings = script["numberOfItems"]
    total_pages = math.ceil(total_listings / 48)
    for product in selector.xpath("//div[@data-search-results-lg]/ul/li[div[@data-appears-component-name]]"):
        link = product.xpath(".//a[contains(@class, 'v2-listing-card')]/@href").get()
        rate = product.xpath(".//span[contains(@class, 'review_stars')]/span/text()").get()
        number_of_reviews = strip_text(product.xpath(".//div[contains(@aria-label,'star rating')]/p/text()").get())
        if number_of_reviews:
            number_of_reviews = number_of_reviews.replace("(", "").replace(")", "")
            number_of_reviews = (
                int(number_of_reviews.replace("k", "").replace(".", "")) * 10
                if "k" in number_of_reviews
                else number_of_reviews
            )
        price = product.xpath(".//span[@class='currency-value']/text()").get()
        original_price = product.xpath(".//span[contains(text(),'Original Price')]/text()").get()
        discount = strip_text(product.xpath(".//span[contains(text(),'off')]/text()").get())
        seller = product.xpath(".//span[contains(text(),'From shop')]/text()").get()
        currency = product.xpath(".//span[@class='currency-symbol']/text()").get()
        data.append(
            {
                "productLink": "/".join(link.split("/")[:5]) if link else None,
                "productTitle": strip_text(
                    product.xpath(".//h3[contains(@class, 'v2-listing-card__titl')]/@title").get()
                ),
                "productImage": product.xpath("//img[@data-listing-card-listing-image]/@src").get(),
                "seller": seller.replace("From shop ", "") if seller else None,
                "listingType": (
                    "Paid listing" if product.xpath(".//span[@data-ad-label='Ad by Etsy seller']") else "Free listing"
                ),
                "productRate": float(rate.strip()) if rate else None,
                "numberOfReviews": int(number_of_reviews) if number_of_reviews else None,
                "freeShipping": (
                    "Yes" if product.xpath(".//span[contains(text(),'Free shipping')]/text()").get() else "No"
                ),
                "productPrice": float(price.replace(",", "")) if price else None,
                "priceCurrency": currency,
                "originalPrice": float(original_price.split(currency)[-1].strip().replace(",", "")) if original_price else "No discount",
                "discount": discount if discount else "No discount",
            }
        )
    return {"search_data": data, "total_pages": total_pages}


async def scrape_search(url: str, max_pages: int = None) -> List[Dict]:
    """scrape product listing data from Etsy search pages"""
    print("scraping the first search page")
    # etsy search pages are dynaminc, requiring render_js enabled
    first_page = await Scraper.async_scrape(
        ScrapeConfig(
            url,
            wait_for_selector="//div[@data-search-pagination]",
            render_js=True,
            auto_scroll=True,
            proxy_pool="public_residential_pool",
            **BASE_CONFIG,
        )
    )
    data = parse_search(first_page)
    search_data = data["search_data"]

    # get the number of total pages to scrape
    total_pages = data["total_pages"]
    if max_pages and max_pages < total_pages:
        total_pages = max_pages

    print(f"scraping search pagination ({total_pages - 1} more pages)")
    # add the remaining search pages in a scraping list
    other_pages = [
        ScrapeConfig(
            url + f"&page={page_number}",
            wait_for_selector="//div[@data-search-pagination]",
            render_js=True,
            proxy_pool="public_residential_pool",
            **BASE_CONFIG,
        )
        for page_number in range(2, total_pages + 1)
    ]
    # scrape the remaining search pages concurrently
    async for response in Scraper.concurrent_scrape(other_pages):
        # try:
        data = parse_search(response)
        search_data.extend(data["search_data"])
        # except Exception as e:
        #     print(f"failed to scrape search page: {e}")
        #     pass
    print(f"scraped {len(search_data)} product listings from search")
    return search_data


async def main():
    search_data = await scrape_search(url="https://www.etsy.com/search?q=wood+laptop+stand", max_pages=3)

    # save the results to a json file
    with open("search_data.json", "w", encoding="utf-8") as file:
        json.dump(search_data, file, indent=2, ensure_ascii=False)


if __name__ == "__main__":
    asyncio.run(main())

Here, we define a scrape_search() function to crawl through the search pages by scraping the first search page and then iterating over the desired number of search pages.

The above etsy.com scraping code should scrape three search pages with a total number of 192 product listings. Here is what the scraped data should look like:

Sample output

Our scraping code is now complete, capable of scraping product, shop and search pages. However, our Etsy scraper will likely encounter blocks after sending multiple requests to the website. Let’s examine a solution!

Avoid Esty.com Scraping Blocking

Etsy.com is a heavily protected website that can detect and block automated bots such as web scrapers. For example, let’s attempt to request a search page on etsy.com with a headless browser to minimize the chances of getting blocked:

from playwright.sync_api import sync_playwright

with sync_playwright() as playwight:
    # Lanuch a chrome browser
    browser = playwight.chromium.launch(headless=False)
    page = browser.new_page()
    # Go to leboncoin.fr
    page.goto("https://www.etsy.com/search?q=personalized+gifts")
    # Take a screenshot
    page.screenshot(path="screenshot.png")

Our request has been detected as a bot and we received a CAPTCHA challenge before proceeding to the web page:

Etsy.com web scraping blocking

To avoid etsy.com scraping blocking, we’ll use Webparsers.

Webparsers provides web scraping, screenshot, and extraction APIs for data collection at scale.

• Anti-bot protection bypass – scrape web pages without blocking!
• Rotating residential proxies – prevent IP address and geographic blocks.
• JavaScript rendering – scrape dynamic web pages through cloud browsers.
• Full browser automation – control browsers to scroll, input and click on objects.
• Format conversion – scrape as HTML, JSON, Text, or Markdown.
• Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

For the example using this service, all we need to do is replace our HTTP client with the webparsers client:

import httpx
from parsel import Selector

response = httpx.get("some etsy.com url")
selector = Selector(response.text)

# in WebParsers SDK becomes
from webparsers import WebConfig, webparsersClient, ScrapeApiResponse

webparsers_client = WebparsersClient("Your API key")
result: ScrapeApiResponse = webparsers_client.scrape(ScrapeConfig(
    # some homegate.ch URL
    "https://www.etsy.com/search?q=personalized+gifts",
    # we can select specific proxy country
    country="US",
    # and enable anti scraping protection bypass
    asp=True,
    # allows JavaScript rendering similar to headless browsers
    render_js=True
))
# get the HTML content
html = result.scrape_result['content']
# use the built-in parsel selector
selector = result.selector

FAQ

To wrap up this guide on etsy.com web scraping, let’s examine some frequently asked questions.

Is scraping etsy.com legal?

All the data on etsy.com are publicly available and it’s legal to scrape them as long as you don’t impact the website performance by maintaining your scraping rate reasonable. However, you should pay attention to the GDPR compliance in the EU, which stands against scraping personal data, such as scraping sellers’ personal information on Etsy. Refer to our previous guide on web scraping legality for more details.

Is there a public API for etsy.com?

There are no public Etsy API endpoints available however Etsy.com is straightforward and legal to scrape. You can use the scraper code described in this tutorial to create your own web scraping API.

Are there alternatives for etsy.com?

Yes, for more scrape guides about websites similar to Etsy, refer to our #scrapeguide blog tag.

Web Scraping Etsy.com – Summary

In this article, we explained how to scrape etsy.com, a popular website for hand-crafted and gift products.

We walked through a comprehensive process on how to scrape product and review data from Etsy products, shop and search pages using Python. We have discovered that etsy.com can detect and block web scrapers. And for that, we have used Webparsers to avoid Etsy web scraping blocking.

Legal Disclaimer and Precautions

This tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect and here’s a good summary of what not to do:

• Do not scrape at rates that could damage the website.
• Do not scrape data that’s not available publicly.
• Do not store PII of EU citizens who are protected by GDPR.
• Do not repurpose the entire public datasets which can be illegal in some countries.

Webparsers does not offer legal advice but these are good general rules to follow in web scraping and for more you should consult a lawyer.