Skip to main content

Webparsers.com

Webparsers Homepage
Login
Menu
Menu
Login

How to Web Scrape with HTTPX and Python

10 min read

Table of Contents

HTTPX is a modern, powerful HTTP client library for Python that’s rapidly gaining popularity in web scraping applications. Its standout features include asynchronous client capabilities and HTTP/2 support, making it an excellent choice for efficient data collection.

In this comprehensive guide, we’ll explore what makes Python’s HTTPX exceptional for web scraping and demonstrate how to leverage it effectively in your projects.

Key Takeaways

Master Python web scraping with HTTPX library for modern HTTP/2 support, async requests, and advanced features like proxy rotation and session management.

  • Use HTTPX for modern Python web scraping with HTTP/2 support and better performance than requests library
  • Implement async web scraping with HTTPX for concurrent requests and improved scraping efficiency
  • Handle proxy rotation and user agent management with HTTPX’s built-in configuration options
  • Use HTTPX’s session management for cookie persistence and connection pooling in scraping workflows
  • Apply proper timeout and retry logic with HTTPX for robust scraping applications
  • Build scalable scrapers with HTTPX’s async capabilities for high-performance data collection

Installing httpx

HTTPX is a pure Python package that can be easily installed using the pip console command:

$ pip install httpx

Alternatively, you can install it using the poetry project package manager:

$ poetry init -d httpx
# or
$ poetry add httpx

Getting Started with HTTPX

HTTPX can handle individual requests directly and supports all the common HTTP methods like GET and POST requests. It also provides convenient methods to parse JSON responses as Python dictionaries:

import httpx

# GET request
response = httpx.get("https://httpbin.dev/get")
print(response)
data = response.json()
print(data['url'])

# POST requests
payload = {"query": "foo"}
# application/json content:
response = httpx.post("https://httpbin.dev/post", json=payload)
# or formdata:
response = httpx.post("https://httpbin.dev/post", data=payload)
print(response)
data = response.json()
print(data['url'])

Here we utilized httpx for JSON parsing using the .json() method of the response object. HTTPX includes many convenient shortcuts like this, making it an accessible HTTP client for web scraping tasks.

Working with httpx Client

For web scraping projects, it’s recommended to use an httpx.Client which allows you to apply custom configurations such as headers, cookies, and proxies across the entire session:

import httpx

with httpx.Client(
    # enable HTTP2 support
    http2=True,
    # set headers for all requests
    headers={"x-secret": "foo"},
    # set cookies
    cookies={"language": "en"},
    # set proxxies
    proxies={
        # set proxy for all http:// connections:
        "http": "http://222.1.1.1:8000",
        # set proxy for all https:// connections:
        "https": "http://222.1.1.1:8000",
        # socks5, socks4 and socks4a proxies can be used as well:
        "https": "socks5://222.1.1.1:8000",
    }
) as session:

The httpx client applies configurations to all requests and maintains server-set cookies automatically.

Implementing Asynchronous Requests with httpx

To leverage httpx asynchronously with Python’s asyncio, use the httpx.AsyncClient() object:

import asyncio
import httpx

async def main():
    async with httpx.AsyncClient(
        # to limit asynchronous concurrent connections limits can be applied:
        limits=httpx.Limits(max_connections=10),
        # tip: increase timeouts for concurrent connections:
        timeout=httpx.Timeout(60.0),  # seconds
        # note: asyncClient takes in the same arguments like Client (like headers, cookies etc.)
    ) as client:
        # to make concurrent requests asyncio.gather can be used:
        urls = [
            "https://httpbin.dev/get",
            "https://httpbin.dev/get",
            "https://httpbin.dev/get",
        ]
        responses = asyncio.gather(*[client.get(url) for url in urls])
        # or asyncio.as_completed:
        for result in asyncio.as_completed([client.get(url) for url in urls]):
            response = await result
            print(response)

asyncio.run(main())

Important: When using async with, ensure all connections complete before the statement closes, otherwise you’ll encounter this exception:

RuntimeError: Cannot send a request, as the client has been closed.

As an alternative to the async with statement, you can manage the httpx AsyncClient manually:

import asyncio
import httpx

async def main():
    client = httpx.AsyncClient()

    # do some scraping
    ...

    # close client
    await client.aclose()

asyncio.run(main())

Common HTTPX Issues and Solutions

While HTTPX is an excellent library for Python, there are several common issues you might encounter. Here are the most frequent problems and their solutions:

httpx.TimeoutException

This error occurs when a request exceeds the specified or default timeout duration. Resolve it by increasing the timeout parameter:

httpx.get("https://httpbin.dev/delay/10", timeout=httpx.Timeout(60.0))

httpx.ConnectError

The httpx.ConnectError exception is raised when connection problems are detected, which can be caused by:

  • unstable internet connection.
  • server being unreachable.
  • mistakes in the URL parameter.

httpx.TooManyRedirects

This exception is raised when a request exceeds the maximum allowed redirects.

This issue can stem from problems with the target web server or httpx redirect handling. Fix it by managing redirects manually:

response = httpx.get(
    "https://httpbin.dev/redirect/3",
    allow_redirects=False,  # disable automatic redirect handling
)
# then we can check whether we want to handle redirecting ourselves:
redirect_location = response.headers["Location"]

httpx.HTTPStatusError

This error is raised when using raise_for_status=True parameter and the server returns a status code outside the 200-299 range, such as 404:

response = httpx.get(
    "https://httpbin.dev/redirect/3",
    raise_for_status=True,
)

In web scraping, status codes outside the 200-299 range often indicate that the scraper is being blocked.

httpx.UnsupportedProtocol

This error occurs when the URL protocol is missing or isn’t one of http://, https://, file://, or ftp://. This commonly happens when URLs are missing the https:// prefix.

Adding Retry Logic to HTTPX Requests

HTTPX doesn’t include built-in retry functionality, but it integrates seamlessly with popular Python retry libraries like tenacity (pip install tenacity).

With tenacity, we can implement retry logic for status codes outside the 200-299 range, httpx exceptions, and even response body content checks:

import httpx
from tenacity import retry, stop_after_attempt, wait_fixed, retry_if_exception_type, retry_if_result

# Define the conditions for retrying based on exception types
def is_retryable_exception(exception):
    return isinstance(exception, (httpx.TimeoutException, httpx.ConnectError))

# Define the conditions for retrying based on HTTP status codes
def is_retryable_status_code(response):
    return response.status_code in [500, 502, 503, 504]

# Define the conditions for retrying based on response content
def is_retryable_content(response):
    return "you are blocked" in response.text.lower()

# Decorate the function with retry conditions and parameters
@retry(
    retry=(retry_if_exception_type(is_retryable_exception) | retry_if_result(is_retryable_status_code) | retry_if_result(is_retryable_content)),
    stop=stop_after_attempt(3),
    wait=wait_fixed(5),
)
def fetch_url(url):
    try:
        response = httpx.get(url)
        response.raise_for_status()
        return response
    except httpx.RequestError as e:
        print(f"Request error: {e}")
        raise e

url = "https://httpbin.dev/get"
try:
    response = fetch_url(url)
    print(f"Successfully fetched URL: {url}")
    print(response.text)
except Exception as e:
    print(f"Failed to fetch URL: {url}")
    print(f"Error: {e}")

In this example, we’re using tenacity’s retry decorator to define retry rules for common httpx errors.

Proxy Rotation for Enhanced Retry Logic

When dealing with blocking in web scraping with httpx, proxy rotation can be combined with tenacity’s retry functionality.

Here’s an example of a common web scraping pattern that rotates proxies and headers when scraping is blocked. Our retry logic will:

  • Retry on status codes 403 and 404
  • Retry up to 5 times
  • Wait randomly 1-5 seconds between retries
  • Switch to a random proxy for each retry
  • Change to a random User-Agent header for each retry

Implementation using httpx and tenacity:

import httpx
import random
from tenacity import retry, stop_after_attempt, wait_random, retry_if_result
import asyncio


PROXY_POOL = [
    "http://2.56.119.93:5074",
    "http://185.199.229.156:7492",
    "http://185.199.228.220:7300",
    "http://185.199.231.45:8382",
    "http://188.74.210.207:6286",
    "http://188.74.183.10:8279",
    "http://188.74.210.21:6100",
    "http://45.155.68.129:8133",
    "http://154.95.36.199:6893",
    "http://45.94.47.66:8110",
]
USER_AGENT_POOL = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5",
]


# Define the conditions for retrying based on HTTP status codes
def is_retryable_status_code(response):
    return response.status_code in [403, 404]


# callback to modify scrape after each retry
def update_scrape_call(retry_state):
    # change to random proxy on each retry
    new_proxy = random.choice(PROXY_POOL)
    new_user_agent = random.choice(USER_AGENT_POOL)
    print(
        "retry {attempt_number}: {url} @ {proxy} with a new proxy {new_proxy}".format(
            attempt_number=retry_state.attempt_number,
            new_proxy=new_proxy,
            **retry_state.kwargs
        )
    )
    retry_state.kwargs["proxy"] = new_proxy
    retry_state.kwargs["client_kwargs"]["headers"]["User-Agent"] = new_user_agent


@retry(
    # retry on bad status code
    retry=retry_if_result(is_retryable_status_code),
    # max 5 retries
    stop=stop_after_attempt(5),
    # wait randomly 1-5 seconds between retries
    wait=wait_random(min=1, max=5),
    # update scrape call on each retry
    before_sleep=update_scrape_call,
)
async def scrape(url, proxy, **client_kwargs):
    async with httpx.AsyncClient(
        proxies={"http://": proxy, "https://": proxy},
        **client_kwargs,
    ) as client:
        response = await client.get(url)
        return response

This demonstrates how to implement retry logic that can rotate proxies and user agent strings on each attempt.

First, we define our proxy and user agent pools, then use the @retry decorator to wrap our scrape function with tenacity’s retry logic.

To modify each retry attempt, we use the before_sleep parameter, which updates our scrape function call with new parameters on each retry.

Here’s a sample test run:

async def example_run():
    urls = [
        "https://httpbin.dev/ip",
        "https://httpbin.dev/ip",
        "https://httpbin.dev/ip",
        "https://httpbin.dev/status/403",
    ]
    to_scrape = [scrape(url=url, proxy=random.choice(PROXY_POOL), headers={"User-Agent": "foo"}) for url in urls]
    for result in asyncio.as_completed(to_scrape):
        response = await result
        print(response.json())


asyncio.run(example_run())

Enhanced Web Scraping with Webparsers

The Webparsers API provides a Python SDK that extends HTTPX functionality with advanced features.

Webparsers provides web scraping, screenshot, and extraction APIs for data collection at scale.

  • Anti-bot protection bypass – scrape web pages without blocking!
  • Rotating residential proxies – prevent IP address and geographic blocks.
  • JavaScript rendering – scrape dynamic web pages through cloud browsers.
  • Full browser automation – control browsers to scroll, input and click on objects.
  • Format conversion – scrape as HTML, JSON, Text, or Markdown.
  • Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

All HTTPX functions are supported by the service’s SDK, making migration seamless:

from scrapfly import ScrapeConfig, ScrapflyClient

client = ScrapflyClient(key="YOUR SCRAPFLY KEY")

result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev/get",
    # enable anti-scraping protection (like cloudflare or perimeterx) bypass
    asp=True,
    # select proxy country:
    country="US",
    # enable headless browser
    render_js=True,
))
print(result.content)

# tip: use concurrent scraping for blazing speeds:
to_scrape = [
    ScrapeConfig(url="https://httpbin.dev/get")
    for i in range(10)
]
async for result in client.concurrent_scrape(to_scrape):
    print(result.content)

The SDK can be installed using pip console command and is free to try:

$ pip install scrapfly-sdk

FAQ

To conclude this Python HTTPX guide, let’s address some frequently asked questions about web scraping with HTTPX.

HTTPX vs Requests

Requests is the most popular HTTP client for Python, known for its accessibility and ease of use. It served as inspiration for HTTPX, which is essentially a modern successor to requests featuring contemporary Python capabilities like asyncio support and HTTP/2.

HTTPX vs Aiohttp

Aiohttp was among the first HTTP clients to support asyncio and influenced HTTPX’s development. While these packages are quite similar, aiohttp is more established whereas HTTPX is newer but offers richer features. For web scraping purposes, HTTPX is generally preferred due to its HTTP/2 support.

How to use HTTP2 with httpx?

HTTPX supports HTTP/2, which is recommended for web scraping as it can significantly reduce scraper block rates. HTTP/2 isn’t enabled by default – you must use the http2=True parameter in httpx.Client(http2=True) and httpx.AsyncClient(http2=True) objects.

How to automatically follow redirects in httpx?

Unlike other Python libraries such as requests, HTTPX doesn’t follow redirects by default. To enable automatic redirect following, use the allow_redirects=True parameter in httpx request methods like httpx.get(url, allow_redirects=True) or httpx client objects like httpx.Client(allow_redirects=True)

Summary

HTTPX represents an exceptional new HTTP client library that’s rapidly becoming the standard choice in Python web scraping communities. Its features like HTTP/2 and asyncio support reduce blocking risks while enabling concurrent web scraping operations.

Combined with tenacity, HTTPX makes web resource requests straightforward with robust retry logic including proxy and user agent header rotation capabilities.