Skip to main content

Webparsers.com

Webparsers Homepage
Login
Menu
Menu
Login

What is HTTP Error code 429 Too Many Requests and How to Fix it

11 min read

Table of Contents

HTTP error 429 Too Many Requests is a commonly encountered status code in web automation, web scraping, and API usage.

While it might seem self-explanatory at first, there are many nuances and different causes behind this error. In this article, we’ll examine what HTTP error 429 is, why it occurs, and how to avoid it through examples in Python and Javascript.

Key Takeaways

  • Resolve too many requests errors by implementing exponential backoff retry logic with request spacing, proxy rotation, and fingerprint management to bypass rate limiting mechanisms
  • Implement exponential backoff retry logic with 429 status code detection for rate limiting
  • Configure request spacing and concurrent connection limits to prevent triggering rate limits
  • Use proxy rotation and IP address distribution to bypass IP-based rate limiting
  • Implement fingerprint rotation for User-Agent, headers, and client identification bypass
  • Monitor rate limit headers (Retry-After, X-RateLimit-*) to optimize request timing strategies
  • Handle different rate limiting strategies including IP, User-Agent, and authentication token limits

What is HTTP Error 429 Too Many Requests?

The 429 error code stands for Too Many Requests, which means the client is performing more requests than permitted.

This typically happens when APIs or websites rate limit connections to either prevent server overload or to sell premium access.

To summarize, HTTP error 429 can be caused by:

  • Making too many requests within a given time frame
  • Sending too many concurrent requests

For calculating connection limits, servers usually use some client feature like:

  • IP Address
  • Authentication token
  • Cookies such as session cookie or special token
  • Headers like User-Agent or X- special headers
  • Client fingerprint which can be generated from several client features like headers, IP, JS fingerprint, etc.

To illustrate this better, let’s examine an example server 429 implementation.

Server Implementation of HTTP Error 429

For this example, we’ll use the Flask web framework in Python to create a simple server that returns error code 429 when the client makes too many requests by measuring connections across a few different axes.

We’ll be using the Flask-Limiter library which provides time-based rate limiting for Flask applications:

python

from flask import Flask, request
from flask_limiter import Limiter
from flask_limiter.util import get_remote_address

app = Flask(__name__)
limiter = Limiter(app, key_func=get_remote_address)

# Limit by IP address
@app.route("/ip-limit")
@limiter.limit("1 per minute", key_func=get_remote_address)
def ip_limit():
    return "Accessed /ip-limit."

# Limit by Authorization request header token
@app.route("/auth-limit")
@limiter.limit("1 per minute", key_func=lambda: request.headers.get("Authorization", "no-auth"))
def auth_limit():
    return "Accessed /auth-limit."

# Limit by User-Agent request header
@app.route("/user-agent-limit")
@limiter.limit("1 per minute", key_func=lambda: request.headers.get("User-Agent", "no-ua"))
def user_agent_limit():
    return "Accessed /user-agent-limit."

# Limit by fingerprint (IP + sorted headers)
@app.route("/fingerprint-limit")
@limiter.limit("1 per minute", key_func=lambda: (get_remote_address() + "|" + str(sorted(request.headers.items()))))
def fingerprint_limit():
    return "Accessed /fingerprint-limit."

if __name__ == "__main__":
    app.run(debug=True)

In the above 429 server example, we implement 4 distinct rate limiting strategies to limit client connections to 1 request per minute:

  • /ip-limit limits by the client IP address
  • /auth-limit limits by the Authorization header
  • /user-agent-limit limits by the User-Agent header
  • /fingerprint-limit limits by the client fingerprint, which in this case is just a combination of IP and headers, though in real-world scenarios it can be more complex

Additionally, most servers provide a Retry-After header or special X- family of headers that inform the client of the remaining limits. In this case, Flask limiter adds these headers:

  • X-RateLimit-Limit – The maximum number of requests allowed in the current period
  • X-RateLimit-Remaining – The number of requests remaining in the current period
  • X-RateLimit-Reset – The time (in UTC epoch seconds) when the rate limit will reset

This example illustrates how request limiting works, and now we can look at how to bypass it.

How to Fix HTTP Error 429 Too Many Requests?

HTTP 429 is caused by rate limiting based on client details, so to bypass this limitation we need to:

  • Increase our rate limit if possible
  • Throttle our requests to match the limit
  • Distribute our requests through multiple agents to bypass the limit

Let’s examine each of these strategies and how to implement them.

Increasing Rate Limit

To start, it’s important to note that some limitations aren’t only there to upsell users but can exist to prevent legitimate abuse. In such cases, many APIs offer free rate limit increases if a free auth token is provided.

For example, with Github’s API, just by providing a free Github PAT token, we can raise the request limit from 60 to 5000 requests per hour:

python

import httpx

# 60 requests per hour
response = httpx.get(
    "https://api.github.com/scrapfly/scrapfly-scrapers", 
)
# 5000 request per hour with free Github PAT
response = httpx.get(
    "https://api.github.com/scrapfly/scrapfly-scrapers", 
    headers={"Authorization": f"Bearer {token}"}
)

So, the first step is to check if the API or website you’re working with offers a way to increase the rate limit.

Throttling Requests

Following rate limiting can be more difficult than it seems, especially when working with concurrent or parallel requests.

The most common cause of 429 Too Many Requests error is un-throttled asynchronous requests that all get sent at once, triggering the rate limit immediately.

To avoid this, requests need to be throttled using time-based mechanisms like Leaky Bucket or Token Bucket or some other time-based distribution algorithm.

Fortunately, you don’t need to implement these algorithms yourself as there are plenty of community libraries that handle this for you. Here’s an example throttle strategy in Python:

python

import asyncio
from time import time

import aiometer
import httpx

session = httpx.AsyncClient()


async def scrape(url):
    response = await session.get(url)
    return response


async def run():
    _start = time()
    urls = ["https://httpbin.dev/get" for i in range(10)]
    results = await aiometer.run_on_each(
        scrape, 
        urls,
        max_per_second=1/6,  # here we can set max rate per second; i.e. aprox 10 requests in 1 minute
    )
    print(f"finished {len(urls)} requests in {time() - _start:.2f} seconds")
    return results


if __name__ == "__main__":
    asyncio.run(run())

# will print:
# finished 10 requests in 9.54 seconds

In the above example, we limit the requests to approximately 10 requests per minute, which is a good starting point for most APIs.

Dynamic Throttle

Not all APIs or websites have a static request limit, which means we can’t apply a static throttling limit like we did with our above examples.

Dynamic rate limiting is usually implemented through the Retry-After header (or similar X- non-standard header) which informs the client how long to wait before retrying or making the next request.

Here’s an example of how to handle dynamic throttling by catching the 429 errors and waiting based on the Retry-After header:

python

import httpx
import asyncio

async def fetch_with_retry(url, retries=3):
    """wrap httpx.get with Retry-After retries"""
    async with httpx.AsyncClient() as client:
        for attempt in range(retries):
            response = await client.get(url)

            if response.status_code == 200:
                return response.json()
            
            # Check for 429 or 503 and handle the Retry-After header
            if response.status_code not in {429, 503}:
                response.raise_for_status()  # Raise if it's another error
                continue
            retry_after = response.headers.get("Retry-After")
            if not retry_after:
                # if no Retry-After header, you can try guessing with exponential backoff
                print("Rate limited but no Retry-After header. Using default backoff.")
                await asyncio.sleep(2 ** attempt)  # Exponential backoff if Retry-After isn't provided
                continue
            # Retry-After can be in seconds or HTTP-date format
            try:
                retry_after_seconds = int(retry_after)
            except ValueError:
                retry_after_seconds = (httpx.parse_header_as_http_date(retry_after) - httpx.utils.now())
            print(f"Rate limited. Retrying after {retry_after_seconds} seconds.")
            await asyncio.sleep(retry_after_seconds)  # Non-blocking sleep
    raise Exception("Max retries exceeded")

async def main():
    url = "https://httpbin.dev/status/429"  # This URL simulates a 429 response for testing
    try:
        result = await fetch_with_retry(url)
        print("Request succeeded:", result)
    except Exception as e:
        print(f"Request failed: {e}")

# start asyncio event loop
asyncio.run(main())

In the above example, we catch the 429 error and wait based on the Retry-After header. If the header is not provided, we can try guessing the wait time using exponential backoff.

Note that dynamic request limiting is very difficult to work with when using asynchronous requests, and it’s usually best to limit requests statically to avoid 429 errors.

Distributing Requests

Unfortunately, many APIs don’t provide a way to increase rate limits, and in those cases, we can use the following strategies:

  • Use proxies to distribute requests through multiple IP addresses
  • Use multiple fingerprint agents (different headers, authentication tokens, etc.) to distribute requests through multiple fingerprints

Depending on how the rate limiting is implemented, you might need to use either one or a combination of both strategies.

For example, if the rate limiting is based on IP address, you can use rotating proxies or your own rotating proxy pool. Here’s a quick example:

python

import httpx
import asyncio
import random

# Array of proxy IPs (in http://username:password@host:port format if authentication is needed)
proxy_ips = [
    "http://proxy1.example.com:8080",
    "http://proxy2.example.com:8080",
    "http://proxy3.example.com:8080",
]

# Array of User-Agent strings
user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15",
    "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0",
]

async def scrape(url):
    """"scrape with random proxy and user agent"""
    # Randomly select a proxy and User-Agent
    proxy = random.choice(proxy_ips)
    user_agent = random.choice(user_agents)
    
    # Set up the client with the selected proxy
    async with httpx.AsyncClient(proxies={"http://": proxy, "https://": proxy}) as client:
        headers = {"User-Agent": user_agent}
        response = await client.get(url, headers=headers)
        print(response.json())  # Display the response (you can store or process this as needed)
        return response.json()

# Main function to make multiple requests
async def main():
    url = "https://httpbin.dev/ip"
    tasks = [scrape(url) for _ in range(100)]  # 100 requests
    results = await asyncio.gather(*tasks)
    print("All requests completed.")

# Run the async main function
asyncio.run(main())

In the above example, we use a rotating proxy pool to distribute requests through multiple IP addresses, essentially multiplying the rate limit by the number of proxies we have.

We also randomize the User-Agent header to avoid detection based on connection fingerprinting, though in reality there’s more to fingerprinting than just the User-Agent header.

Power Up with WebParsers

WebParsers has millions of proxies and connection fingerprints that can be used to bypass rate limits and significantly simplify your web automation projects.

WebParsers provides web scraping, screenshot, and extraction APIs for data collection at scale. Each product is equipped with automatic bypass for any anti-bot system, achieved through:

  • Maintaining a fleet of real, reinforced web browsers with authentic fingerprint profiles
  • Millions of self-healing proxies with the highest possible trust score
  • Constantly evolving and adapting to new anti-bot systems

FAQ

Before we finish, let’s look at some common questions about HTTP error 429 Too Many Requests:

What is the difference between 429 and 401 status codes?

The 429 status code means Too Many Requests and is used when the client is making more requests than allowed. The 401 status code means Unauthorized and is used when the client is not authorized to access the resource or is simply being blocked.

How to know what’s the rate limit of an API or a website?

APIs or websites generally return a 429 status code when the rate limit is exceeded. This response usually also contains headers like X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset that inform the client of the remaining limits, or a Retry-After header that tells the client how long to wait before retrying.

Can I get blocked for getting 429 status code?

Yes, too many 429 status code requests can lead to getting blocked by the website or API. It’s important to respect the rate limits set by the service to avoid getting blocked or causing harm to the service if the limits are set to reasonable values.

Summary

To summarize, HTTP error 429 Too Many Requests is a common status code that can be caused by making too many requests within a given time frame or sending too many concurrent requests.

To fix HTTP error 429, we can:

  • Increase our rate limit if the websites or APIs offer a way to do so
  • Throttle our requests to match the limit
  • Distribute our requests through multiple agents like proxies and client fingerprints (such as the User-Agent header)

Finally, it’s important to note that rate limiting often serves a purpose, and it’s essential to respect the limits set by the website or API to avoid getting blocked or causing harm to the service.