How to Rotate Proxies for Python Web Scraping

Scraping at scale means one thing above all: don't send too many requests from one IP. Proxy rotation spreads your traffic across many addresses so no single one trips a rate limit. Here's how to do it cleanly in Python.

Option 1: a provider-side rotating endpoint

The simplest approach is to let the proxy provider rotate for you. With a rotating residential proxy you point every request at one endpoint and receive a different exit IP automatically.

import requests

proxies = {
    "http": "http://user:[email protected]:8000",
    "https": "http://user:[email protected]:8000",
}

r = requests.get("https://example.com", proxies=proxies, timeout=20)
print(r.status_code)

Every call through that endpoint lands on a fresh residential IP — no rotation logic in your code.

Option 2: rotate a list yourself

If you manage a list of proxies, pick a different one per request:

import random, requests

pool = [
    "http://user:pass@ip1:port",
    "http://user:pass@ip2:port",
    "http://user:pass@ip3:port",
]

def fetch(url):
    proxy = random.choice(pool)
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=20)

Go faster with async

For large jobs, aiohttp lets you fetch many URLs concurrently, each through the rotating endpoint:

import aiohttp, asyncio

PROXY = "http://user:[email protected]:8000"

async def fetch(session, url):
    async with session.get(url, proxy=PROXY) as resp:
        return await resp.text()

async def main(urls):
    async with aiohttp.ClientSession() as session:
        return await asyncio.gather(*(fetch(session, u) for u in urls))

Best practices

• Add randomised delays between requests to mimic human pacing.
• Send realistic browser headers, including a current User-Agent.
• Use sticky sessions for multi-step flows like logins — see rotating vs sticky sessions.
• Retry failed requests on a fresh IP with exponential backoff.
• Geo-target when scraping localised data via proxy locations.

Choosing the right pool

For protected targets use rotating residential; for simple public pages, cheaper datacenter proxies are faster. Not sure? Read how to choose the right proxy for web scraping.