How to Rotate Proxies in Python Using Requests (Easy Guide)

When performing web scraping, target websites frequently ban IP addresses. This issue becomes more prevalent when dealing with sites that implement sophisticated anti-bot solutions from providers like Cloudflare, Google, and Akamai.

This means you must utilize proxies to conceal your actual IP address. Since these proxies may also face bans from target websites, implementing regular proxy rotation becomes crucial.

In this web scraping guide, you will discover two approaches to implement and rotate Python proxies: utilizing Webparsers (the straightforward method) and employing Requests in Python (the more complex approach).

Rotating Proxies with Webparsers and Python (the Easy Way)

Utilizing Webparsers represents the most straightforward approach to implement and rotate proxies with Python (Requests). Maintaining a pool of unblocked proxies for web scraping presents ongoing challenges. Webparsers eliminates this proxy rotation burden, allowing you to concentrate on data extraction rather than infrastructure management.

Simply sign up for Webparsers and obtain 5000 complimentary API credits. Then, retrieve your API key from the dashboard:

Getting Started with Webparsers

You can then implement the Webparsers proxy using Requests like this:

import requests 

proxies = { "http": "http://api.webparsers.com:APIKEY@proxy-server.scraperapi.com:8001", "https": "http://api.webparsers.com:APIKEY@proxy-server.scraperapi.com:8001" } 

r = requests.get('http://httpbin.org/ip', proxies=proxies, verify=False) 

print(r.text)

Each time you send a request through the Webparsers proxy, the service automatically rotates proxies and assigns a fresh one for every request. The process couldn’t be more streamlined!

With Webparsers handling your web scraping needs, you can delegate all proxy sourcing, validation, and rotation complexities to the service while focusing on your core business logic. You can also be confident in having fresh proxies available consistently, as Webparsers maintains access to over 40 million proxies!

Rotating Proxies with Python (the Traditional, More Complex Way)

Now, let’s examine a more conventional approach to proxy rotation in Python Requests. As you’ll discover, this method involves significantly more complexity and demands considerable time and attention to maintain smooth operation.

Step 1. Setting up the Prerequisites of Python Request Proxies

Ensure Python is installed on your system. Python version 3.7 or higher works for this tutorial. Create a new directory to store all project code and establish an app.py file within:

$ mkdir proxy_rotator
$ cd proxy_rotator
$ touch app.py

You also need requests installed. Install it easily via PIP:

$ pip install requests

Step 2. How to Source a Proxy List?

Before implementing proxy rotation, you need a proxy list. Various lists exist online, including both paid and free options. Each comes with distinct advantages and disadvantages.

Free Proxy List serves as a popular source for complimentary proxies. The primary concern with free proxy lists is that target websites may have already blocked most proxies, requiring testing to verify proxy functionality.

You can download proxy lists from Free Proxy List into a txt file.

Note: If you select the straightforward Webparsers method described earlier, you’ll appreciate that Webparsers continuously monitors all proxies to ensure target websites haven’t blocked them!

Step 3. Making a Request Without a Proxy

Let’s begin by examining how to make requests using the requests library without proxies. You can accomplish this through two methods: directly using the requests.get (or similar) method, or creating a Session and using it for requests.

Direct requests using requests.get work like this:

import requests 
html = requests.get("https://yasoob.me")
print(html.status_code)
# output: 200

The same request using Session works like this:

import requests 
s = requests.Session() 
html = s.get("https://yasoob.me") 
print(html.status_code) 
# Output: 200

Discussing both methods is important since proxy implementation differs slightly between them.

Step 4. Using a Proxy with Requests

Using proxies with requests is straightforward. Simply provide requests with a dictionary containing HTTP and HTTPS keys along with their corresponding proxy URLs. You may use identical proxy URLs for both protocols.

Note: Since this article uses free proxies, the proxy URLs in code blocks might become non-functional by the time you read them. Follow along by replacing proxy URLs in code samples with working proxies from Free Proxy List.

Here’s sample code for proxy usage in requests without creating a Session object:

import requests 
proxies = { 'http': 'http://47.245.97.176:9000', 'https': 'http://47.245.97.176:9000', } 

response = requests.get('https://httpbin.org/ip', proxies=proxies) 

print(response.text) 

# Output: { # "origin": "47.245.97.176" # }

Here’s the same example using the Session object:

import requests 
proxies = { 'http': 'http://47.245.97.176:9000', 'https': 'http://47.245.97.176:9000', } 

s = requests.Session() 

s.proxies = proxies 

response = s.get('https://httpbin.org/ip') 

print(response.text) 

# Output: { # "origin": "47.245.97.176" # }

CERTIFICATE_VERIFY_FAILED SSL errors commonly occur when using free proxies. The error appears as follows:

requests.exceptions.SSLError: HTTPSConnectionPool

(host='httpbin.org', port=443): 

Max retries exceeded with url: /ip (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))

You can resolve this error by passing verify=False to the get method:

requests.get('https://httpbin.org/ip', proxies=proxies, verify=False) 

# or 

s.get('https://httpbin.org/ip', verify=False)

Step 5. Using an Authenticated Proxy with Requests

Implementing authenticated proxies with requests is equally simple. Modify the proxies dictionary to include username and password for each proxy URL:

proxies = { 'http': 'http://username:password@proxy.com:8080', 

https': 'http://username:password@proxy.com:8081', }

Replace username and password with functional credentials. The remaining request code stays identical to previous samples.

Step 6. Setting a Proxy Via Environment Variables

You can also utilize proxies without adding proxy-specific code to Python. This works by setting appropriate environment variables. The requests library honors HTTP_PROXY and HTTPS_PROXY environment variables. When set, requests uses their values as corresponding proxy URLs.

Set these environment variables in Unix-like systems by opening the terminal and entering:

export HTTP_PROXY='http://47.245.97.176:9000' 

export HTTPS_PROXY='http://47.245.97.176:9000'

Now you can remove proxy-specific code from your Python program, and it will automatically utilize the proxy endpoint configured via environment variables!

Test this by running the following code and ensuring output matches the proxy endpoint set through environment variables:

import requests 

response = requests.get('https://httpbin.org/ip', proxies=proxies) 

print(response.text) 

# Output: { # "origin": "47.245.97.176" # }

Step 7. Rotating Proxies with Each Request

As mentioned in the introduction, proxies can also face blocking. Therefore, implementing proxy rotation and avoiding single proxy usage for multiple requests becomes essential. Let’s examine how to rotate proxies in Python using requests.

Step 7.1. Loading proxies from a proxy list

To begin, save proxies from Free Proxy List into a proxy_list.txt file within the proxy_rotator directory. The file will appear like this:

196.20.125.157:8083 
47.245.97.176:9000 
54.39.132.131:80 
183.91.3.22:11022 
154.236.179.226:1981 
41.65.46.178:1981 
89.175.26.210:80 
61.216.156.222:60808 
115.144.99.220:11116 
... 167.99.184.232:3128

Open the app.py file and write this code to load proxies into a list:

def load_proxy_list(): 
with open("proxy_list.txt", "r") as f: 

proxy_list = f.read().strip().split() return proxy_list

Step 7.2. Verify the proxy works

With a proxy list available, testing all proxies for functionality and removing non-working ones becomes important. Test this by sending requests to httpbin via the proxy and ensuring responses contain the proxy IP. If requests fail, discard the proxy.

You can make the discarding process more refined by ensuring request failures result from proxy issues rather than unrelated network problems. For simplicity, let’s discard proxies whenever any error (exception) occurs. Here’s code that accomplishes this:

def check_proxy(proxy_string): 
proxies = { 'http': f'http://{proxy_string}', 'https': f'http://{proxy_string}', } 

try: 
response = requests.get('https://httpbin.org/ip', proxies=proxies, timeout=30) 

if response.json()['origin'] == proxy_string.split(":")[0]: 

# Proxy works return True 

# Proxy doesn't work return False except Exception: return False

The code is straightforward: pass a proxy string (e.g., 0.0.0.0:8080) to check_proxy as an argument, then check_proxy sends requests to httpbin.org/ip through the specified proxy. If responses contain the proxy IP, it returns True; otherwise (or if requests fail), it returns False. The code includes timeout definitions for each request. If responses aren’t received within defined timeouts, exceptions raise, ensuring you avoid slow proxies.

Step 7.3. Rotating the proxy with each request

You can now combine functions from the previous two code listings to rotate proxies with each request. Here’s one potential approach:

from random import choice 

def get_working_proxy(): 
   random_proxy = choice(proxy_list) 
   while not is_proxy_working(random_proxy): 
   proxy_list.remove(random_proxy) 
   random_proxy = choice(proxy_list) 

return random_proxy 

def load_url(url): 
   proxy = get_working_proxy() 
   proxies = { 'http': f'http://{proxy}', 'https': f'http://{proxy}', } 
   response = requests.get(url, proxies=proxies) 
   # parse the response 
   # ... return response.status_code 

urls_to_scrape = [ "https://news.ycombinator.com/item?id=36580417", "https://news.ycombinator.com/item?id=36575784", "https://news.ycombinator.com/item?id=36577536", # ... ] 

proxy_list = load_proxy_list() 

     for url in urls_to_scrape: 
        print(load_url(url))

Let’s analyze this code. It contains a get_working_proxy() function that selects random proxies from the proxy list, verifies functionality, then returns them. If proxies don’t work as expected, the function removes them from the proxy list. The load_url() function obtains working proxies by calling get_working_proxy() and uses returned proxies to route requests to target URLs. Finally, code initiates the scraping process. The important aspect is that random proxies are used for each request, helping distribute scraping load across multiple proxies.

How to Improve the Proxy Rotator

Many approaches exist to enhance the basic proxy rotator you’ve created. First, revise exception handling code to ensure proxy discarding occurs only when they’re actually faulty.

Another improvement involves rechecking discarded proxies periodically. Generally, free proxies alternate between working and non-working states frequently. You can also add logic to load proxies directly from the Free Proxy List website instead of manually saving them to txt files first.

Maximize Web Scraping Success with Effective Proxy Use and Rotation

You’ve learned how to implement proxies with Requests in Python, source, verify, and rotate them. Now you might wonder about the optimal proxy method to use.

While you can choose traditional approaches, prepare for frequent code adjustments and constant proxy updates. This can become time-consuming and disrupt data collection workflows. The optimal approach involves using tools that handle proxy rotation automatically, enabling quick and large-scale data acquisition.