What Is a 429 Error and How Do Scrapers Handle Rate Limiting
HTTP 429 Too Many Requests is the most common failure mode in web scraping. Unlike a parser break or a structural change — which can be subtle — a 429 is an explicit signal from the server: you are requesting too fast. The response tells you exactly what happened. The challenge is knowing what to do next, and how to prevent it from degrading your pipeline coverage silently.
This article explains what causes 429 errors, how they differ from related HTTP status codes, and what infrastructure choices make scraping pipelines resilient to rate limiting. If you are building on our API Marketplace, rate limiting is handled at the infrastructure level — see the API Docs for per-endpoint rate limit details. For a broader view of anti-bot challenges, see our article on handling anti-bot systems.
What HTTP 429 Means and Related Status Codes
Rate limiting is not always a single status code. Several HTTP responses signal access restrictions, and handling them differently is important for pipeline reliability:
| Status code | Meaning | Retry-After header | Typical source |
|---|---|---|---|
| 429 | Too Many Requests — application rate limit exceeded | Often present | Application server, API gateway |
| 1015 | Cloudflare rate limited — CDN-level block | No | Cloudflare CDN |
| 503 | Service Unavailable — capacity or anti-bot challenge | Sometimes | Server overload or anti-bot system |
| 403 | Forbidden — IP or session blocked | No | Anti-bot system, IP block |
| 499 | Client closed request — timeout before server responded | No | Slow or throttled server response |
429 is the cleanest signal — the server is telling you it received your request but will not process it. Cloudflare’s 1015 behaves similarly but fires at the CDN layer before the request reaches the application. A 403 after repeated requests usually signals an IP ban rather than a temporary rate limit — the correct response is proxy rotation, not waiting.
Why 429 Errors Are the Primary Scraping Failure Mode
Rate limiting operates at multiple levels simultaneously in modern web infrastructure:
- IP-level rate limits. The most common trigger. Once a single IP makes more than N requests in a time window, the server starts returning 429s for that IP. Without proxy rotation, a scraper hits this ceiling quickly on any high-volume source. See our article on proxy management for how to distribute requests across IP pools.
- Session-level rate limits. Some platforms track request rate per session or cookie, not just per IP. Rotating IPs without rotating sessions partially addresses this — full session management is required.
- API quota limits. Official APIs implement quota systems — requests per minute, requests per day, requests per API key. These are distinct from anti-bot rate limiting: they are by design and require quota management, not evasion. See our article on scraping APIs vs HTML for the difference.
- Behavioral triggers. Advanced anti-bot systems issue 429s not purely based on volume but on behavioral anomalies — perfectly regular request intervals, non-human navigation patterns, missing HTTP headers. These require behavioral calibration beyond simple pacing.
Strategies for Handling 429 Errors in Scrapers
| Strategy | What it addresses | Implementation | Tradeoff |
|---|---|---|---|
| Respect Retry-After header | Application-level rate limits with explicit wait time | Parse header, pause for stated duration before retry | Simple but slows collection during limit windows |
| Exponential backoff | Burst-triggered limits without Retry-After | Double wait time on each successive 429, up to a ceiling | Reduces throughput; must cap maximum wait or jobs stall |
| Proxy rotation | IP-level rate limits | Switch IP on 429; distribute load across pool | Pool size determines how long you can sustain volume |
| Request pacing | Prevents triggering rate limits in the first place | Insert randomised delays between requests per source | Reduces peak throughput; must balance speed vs stability |
| Per-source rate budgets | Sustained high-volume collection across many sources | Define max req/min per source; enforce in job scheduler | Requires source-level configuration and scheduler support |
The most important rule: never immediately retry a 429 at the same rate. Repeated 429s from the same IP without backing off often escalate to a longer-term block — turning a temporary rate limit into a 403. See our article on scraping at scale for how rate management fits into large-scale pipeline design.
Monitoring 429 Rates as a Pipeline Health Signal
A rising 429 rate on a source is an early warning before a full block. Tracking 429 frequency per source — not just total error count — allows proactive response:
- If the 429 rate on a source rises above 10–15% of requests, the collection approach needs adjustment before the rate escalates further.
- A sudden spike in 429s after previously stable collection usually signals an anti-bot update on the target platform, not a traffic increase.
- Sustained 429s that do not resolve after backoff point to an IP-level ban, requiring proxy rotation rather than waiting.
See our article on scraping monitoring and alerting for how to build 429 rate tracking into pipeline health dashboards.
How Webparsers Handles Rate Limiting
- Per-source rate budgets are configured from day one. Each source has a defined request rate based on its known tolerance. We do not apply a single global rate across all sources — limits are calibrated per target.
- 429 responses trigger adaptive pacing, not immediate retry. When a source returns 429s above a threshold, collection automatically slows and switches to a different IP before retrying. Persistent 429s escalate to an alert, not a retry loop.
- Our API Marketplace endpoints abstract rate limit management. For data available through API Marketplace, rate limiting is handled internally — clients request data and receive it without managing per-source rate limits themselves.
- 429 rate is a monitored metric, not just an error log entry. We track 429 frequency per source on a rolling basis. Rising rates trigger investigation before they become coverage gaps.
- We distinguish rate limiting from IP bans. A 429 that resolves after backoff is rate limiting. A 403 that persists after proxy rotation is a ban. The response is different for each, and confusing them wastes time on the wrong fix.
Discuss Your Pipeline Rate Limit Challenges
Frequently Asked Questions
What does HTTP status code 429 mean?
429 Too Many Requests means the server received more requests from your client than its rate limit allows in a given time window. It is defined in RFC 6585. The server may include a Retry-After header indicating how long to wait before retrying. It is a temporary condition — unlike a 403 (forbidden), a 429 does not mean you are permanently blocked, only that you need to slow down.
How do you fix a 429 error in a web scraper?
The correct responses are: parse and respect the Retry-After header if present; implement exponential backoff for retries; reduce request frequency on the affected source; and rotate proxy IPs to distribute the request load across multiple addresses. Never immediately retry at the same rate — repeated 429s without backing off can escalate to a longer-term IP block.
What is Cloudflare error 1015?
Error 1015 is Cloudflare’s rate limiting response, returned at the CDN layer before the request reaches the origin server. It behaves similarly to a 429 but does not always include a Retry-After header. The correct response is the same: back off and rotate to a new IP. Cloudflare 1015s are more common on sites with aggressive CDN-level rate limiting, and residential proxies are typically more effective than datacenter IPs for getting through.
Is there a difference between a 429 error on an API and on a website?
Yes. An API 429 is usually a quota limit — the platform allows a defined number of requests per minute or day per API key, and you have exceeded it. The fix is quota management (batch requests, add API keys, upgrade the plan). A website 429 is usually anti-bot rate limiting — the platform is detecting abnormal traffic volume. The fix is pacing, proxy rotation, and behavioral calibration. The two require different responses.