Skip to main content

Webparsers.com

Webparsers Homepage
Login
Menu
Menu
Login

What is HTTP 406 Error? (Not Acceptable)

4 min read

Table of Contents

When working on web scraping or automation, encountering HTTP errors can be frustrating, and HTTP error 406 is one that indicates a mismatch in the type of content being requested.

In this article, we’ll explore what HTTP 406 means, the common causes behind it, and whether it could be used as a blocking strategy. We’ll also dive into how Webparsers can help you bypass this error effectively.

Key Takeaways

Fix 406 not acceptable errors by configuring proper Accept headers (text/html, application/xhtml+xml), language headers (en-US,en;q=0.9), and encoding support (gzip, deflate, br) to successfully access web content.

  • HTTP 406 “Not Acceptable” occurs when server cannot deliver content matching the client’s Accept- headers
  • Common causes include misconfigured Accept, Accept-Language, or Accept-Encoding headers in requests
  • Header configuration is crucial – use realistic browser headers like text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
  • Language headers should match target region (e.g., en-US,en;q=0.9 for US English content)
  • Encoding support should include gzip, deflate, br to match modern browser capabilities
  • Content negotiation allows servers to return different formats based on client preferences
  • Anti-blocking strategy – 406 errors can sometimes be used as a blocking mechanism by websites
  • Webparsers integration provides built-in header management and proxy rotation to avoid 406 blocks

What is HTTP Error 406?

406 Not Acceptable error occurs when the server cannot deliver a response in a format that matches the criteria specified by the client’s Accept- headers. In essence, while the server comprehends the request, it cannot locate a response that conforms to the content types or formats the client is prepared to accept.

What are HTTP 406 Error Causes?

The primary cause of a 406 error stems from improperly configured Accept- headers. These headers communicate to the server what content types the client anticipates in the response, including:

  • Accept: Specifies the expected media type, like application/json or text/html.
  • Accept-Language: Indicates the preferred languages for the response, e.g., en-US.
  • Accept-Encoding: Defines the compression formats that the client can handle, like gzip or deflate.

When the server is unable to provide a response that aligns with the specified Accept- headers, it will return a 406 status code.

Practical Example

Let’s explore how to configure headers, specifically Accept- headers, in common tools like python’s httpx library, and cURL.

cURL
Python (httpx)
Javascript (fetch)
Rust
Go
Ruby (typhoeus)
PHP (guzzle)

curl -H "Accept: application/json" -H "Accept-Language: en-US" https://httpbin.dev/json

In both examples, the client is requesting a response in application/json format and prefers the response language in en-US. If the server cannot match these criteria, a 406 error might occur.

To prevent 406 errors, ensure that your Accept- headers are configured correctly for the resource you’re attempting to access.

406 in Web Scraping

In web scraping scenarios, the 406 status code is frequently encountered when Accept- family headers are absent or incorrectly configured.

Most HTTP clients do not include default Accept- headers, requiring manual configuration. To determine which headers are necessary, observe how the website functions in your web browser using Browser Developer Tools. Through the Network tab, you can examine the precise Accept headers your browser transmits and reproduce them in your scrapers.

Alternatively, there’s a minor possibility that the 406 error is deliberately returned by the server to obstruct web scraping and mislead the scraper into believing there’s a technical problem. In such cases, refer to our guide on fortifying web scrapers against blocking.

Power Up with Webparsers

Webparsers provides web scraping, screenshot, and extraction APIs for data collection at scale.

  • Anti-bot protection bypass – scrape web pages without blocking!
  • Rotating residential proxies – prevent IP address and geographic blocks.
  • JavaScript rendering – scrape dynamic web pages through cloud browsers.
  • Full browser automation – control browsers to scroll, input and click on objects.
  • Format conversion – scrape as HTML, JSON, Text, or Markdown.
  • Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

scrapfly middleware

It takes Webparsers several full-time engineers to maintain this system, so you don’t have to!

Summary

HTTP 406 errors result from a mismatch between the Accept- headers transmitted by the client and the formats the server can provide. Although uncommon, these errors can occasionally serve as a blocking mechanism. Using Webparsers’ advanced tools, including proxy rotation and customizable requests, you can circumvent 406 blocks and maintain smooth web scraping operations.