Web Scraping Scroll Down and Infinite Scroll

Web Scraping Scroll Down and Infinite Scroll

A significant portion of modern websites do not load their full content in the initial HTML response. Social media feeds, e-commerce category pages, news aggregators, and job listing sites use infinite scrolling or lazy loading — deferring content until the user scrolls it into view. For a scraper that sends a simple HTTP request and parses the response, this means it captures only the first screen of content and misses everything below.

Handling scroll-dependent content requires a real browser engine — Playwright or Puppeteer — to execute the JavaScript that triggers content loading as scroll position changes. This article covers the scroll patterns you will encounter, how to implement scroll automation in Python and JavaScript, how to detect when all content has loaded, and when a direct API approach is more efficient than scroll simulation. Webparsers builds data collection pipelines for scroll-heavy targets including social media, e-commerce, and job boards — see our API Marketplace for available data endpoints.

Talk to a Data Engineer

Scroll-Dependent Content Patterns

Before Implementing Scroll: Check for a Direct API

Scroll automation in a headless browser is slower, more resource-intensive, and more fragile than direct HTTP requests. Before implementing scroll, inspect the target page's network traffic in Chrome DevTools (F12 → Network tab → filter by XHR/Fetch) while manually scrolling. Look for API calls that load the additional content. Many infinite scroll implementations make requests to patterned API endpoints:

GET /api/posts?page=2&limit=20
GET /api/feed?cursor=eyJpZCI6MTIzfQ==&count=25
GET /search/results?offset=40&q=keyword

If the API accepts a page number, offset, or cursor parameter, you can iterate through all pages with direct HTTP requests — much faster than browser automation, and easier to maintain. Only implement scroll automation when the content loading mechanism cannot be replicated with direct API calls.

Scroll Automation with Playwright (Python)

Basic Infinite Scroll: Scroll Until No Height Change

import asyncio
from playwright.async_api import async_playwright

async def scroll_to_bottom(page, pause_ms=1500, max_scrolls=50):
    """Scroll to bottom repeatedly until page height stops growing."""
    previous_height = 0
    scroll_count = 0

    while scroll_count < max_scrolls:
        current_height = await page.evaluate("document.body.scrollHeight")

        if current_height == previous_height:
            break  # No new content loaded — done

        previous_height = current_height
        await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        await page.wait_for_timeout(pause_ms)  # Wait for XHR to complete
        scroll_count += 1

    return scroll_count

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto("https://example.com/feed", wait_until="domcontentloaded")

        scrolls = await scroll_to_bottom(page)
        print(f"Scrolled {scrolls} times")

        # Extract content after all items are loaded
        items = await page.query_selector_all(".feed-item")
        for item in items:
            text = await item.inner_text()
            print(text)

        await browser.close()

asyncio.run(main())

Key points: the pause_ms delay after each scroll must be long enough to allow the XHR request to complete and the DOM to update before the next height check. Too short a delay causes the loop to exit early, treating an in-flight request as a no-content signal. max_scrolls prevents infinite loops on pages where height calculation behaves unexpectedly.

Incremental Scroll: Loading Lazy Elements

Some pages use lazy loading for elements that are present in the DOM but not yet rendered — images with loading="lazy", or fields populated by a viewport intersection observer. For these, scrolling incrementally through the page (rather than jumping directly to the bottom) ensures each element enters the viewport and triggers its load:

async def scroll_incrementally(page, step_px=400, pause_ms=300):
    """Scroll down in steps to trigger lazy-loading elements."""
    total_height = await page.evaluate("document.body.scrollHeight")
    current_pos = 0

    while current_pos < total_height:
        await page.evaluate(f"window.scrollTo(0, {current_pos})")
        await page.wait_for_timeout(pause_ms)
        current_pos += step_px
        # Re-check total height in case new content was appended
        total_height = await page.evaluate("document.body.scrollHeight")

Load More Button: Click Until Absent

async def click_load_more(page, button_selector="button.load-more", pause_ms=1500):
    """Click 'Load More' button until it disappears."""
    while True:
        button = await page.query_selector(button_selector)
        if not button:
            break  # No more button — all content loaded
        await button.click()
        await page.wait_for_timeout(pause_ms)

Scroll Automation with Puppeteer (JavaScript)

const puppeteer = require('puppeteer');

async function scrollToBottom(page, pauseMs = 1500, maxScrolls = 50) {
    let previousHeight = 0;
    let scrollCount = 0;

    while (scrollCount < maxScrolls) {
        const currentHeight = await page.evaluate(() => document.body.scrollHeight);

        if (currentHeight === previousHeight) break;

        previousHeight = currentHeight;
        await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
        await new Promise(r => setTimeout(r, pauseMs));
        scrollCount++;
    }
    return scrollCount;
}

(async () => {
    const browser = await puppeteer.launch({ headless: true });
    const page = await browser.newPage();
    await page.goto('https://example.com/feed', { waitUntil: 'domcontentloaded' });

    const scrolls = await scrollToBottom(page);
    console.log(`Scrolled ${scrolls} times`);

    const items = await page.$$eval('.feed-item', els => els.map(el => el.innerText));
    console.log(items);

    await browser.close();
})();

Common Scroll Scraping Problems and Fixes

How Webparsers Handles Scroll-Dependent Collection in Production

1. We identify the content loading mechanism before writing scroll logic. For each target, we inspect network traffic during manual scrolling to determine whether content loads via scroll-triggered XHR (and if so, whether the API endpoint is directly accessible), a Load More button, viewport-triggered lazy loading, or a combination. This determines the collection approach — direct API iteration where possible, browser scroll automation only when required. See our API Docs and API Marketplace.
2. We use Playwright with residential proxies for browser-automated scroll collection. Scroll automation with a headless browser on anti-bot-protected targets requires residential IP addresses to pass IP reputation checks, and realistic scroll timing and request patterns to avoid behavioural detection. Fixed-interval scrolling (constant speed, constant step size) is a detectable automation signal; we use randomised delays and speed profiles. See our articles on headless browsers for scraping and proxy management.
3. We handle scroll termination correctly for each target's loading behaviour. Height-comparison termination works for most infinite scroll implementations. For targets where page height grows unreliably, we track item count before and after each scroll to detect when no new items were appended. Scroll logic is wrapped in maximum iteration limits and timeout guards to prevent collection jobs from hanging on unusual page states.
4. We deduplicate and validate collected records before delivery. Scroll-based collection often produces duplicate records — items visible across multiple scroll positions, or items returned by overlapping XHR requests. We deduplicate on unique identifiers (post ID, product ID, listing URL) before delivery. Incomplete records (items that started loading but did not fully render before extraction) are detected by required field validation and re-queued rather than delivered as partial records. See our article on data normalization and enrichment.
5. We configure retry logic for scroll jobs interrupted by rate limiting or detection. A scroll session that is rate-limited or flagged mid-way through a deep feed produces a partial dataset. We detect interruption signals (content stops loading before expected end, CAPTCHA served, redirect to login page) and retry from a fresh session with a new proxy rather than delivering the partial result. See our article on data delivery and integration.

Discuss Your Data Collection Requirements

Frequently Asked Questions

Why do web scrapers need to scroll down?

Many modern websites load content dynamically as the user scrolls — infinite scrolling or lazy loading. The initial HTML response contains only the first screen of content; additional items are fetched via JavaScript as the page is scrolled. A scraper that only fetches the initial HTML captures a partial dataset. Scroll automation using a headless browser triggers the same JavaScript that loads additional content, making all items accessible for extraction.

What is the difference between infinite scroll and lazy loading?

Infinite scrolling continuously appends new content to the page as the user reaches the bottom — social media feeds and e-commerce category pages commonly use this. Lazy loading defers loading of specific elements (images, price fields) until they scroll into the viewport, improving initial load performance. Both require scroll simulation in a headless browser to trigger the JavaScript that loads the deferred content.

Can I scrape infinite scroll pages without a headless browser?

Sometimes. Inspect the network tab in DevTools while manually scrolling — look for XHR or Fetch requests that load the additional content. If the underlying API accepts page number, offset, or cursor parameters, you can make direct HTTP requests to that endpoint instead of simulating scroll, which is faster and more reliable. Only implement browser scroll automation when the content loading cannot be replicated with direct API calls.

How do I stop scrolling when all content is loaded?

Compare page height before and after each scroll action. If scrolling produced no change in page height, all content has been loaded. Always wait after scrolling to allow in-flight XHR requests to complete before checking height. Set a maximum iteration limit as a safety stop to prevent infinite loops on pages where height calculations behave unexpectedly.

How does Webparsers handle infinite scroll in data pipelines?

Webparsers uses Playwright-based headless browser automation for scroll-dependent targets. Collection approach is chosen per target: direct XHR API calls where the underlying data endpoint is accessible, browser scroll automation for targets that require it. Scroll jobs use residential proxies, randomised timing, item-count termination detection, deduplication, and retry logic for interrupted sessions.