Web Scraping Proxies: Rotating Residential IPs from US$1.50/GB

Why web scraping needs proxies

Sites cap how many requests one IP may send in a period. Past the cap they answer HTTP 429 Too Many Requests, often with a Retry-After header saying how long to wait — and a crawler on one server IP reaches it long before a catalogue is done.

Blocks also come by network, not just by address: Cloudflare lets site owners block or challenge traffic by IP, by ASN or by country. A datacenter IP shares its ASN with every other server at the same host, while residential exits sit on consumer ISP networks.

Automated traffic is now the majority — Cloudflare reported in July 2026 that more than half of Internet traffic is non-human, and since July 2025 it blocks AI training crawlers by default on new domains — so sites keep tightening their rules. Collect politely: follow robots.txt and the site's terms, keep a steady pace and never fetch the same page twice when once will do.

• MDN — 429 Too Many Requests
• Cloudflare docs — IP Access rules (by IP, ASN or country)
• Cloudflare blog — Content Independence Day, one year on (July 2026)

Recommended setup for web scraping

Scroll horizontally with the arrow keys to read all content.

The button below opens Rotating Residential Proxies with the AI and automation use case. Switch Line to Budget for US$1.50/GB before you buy traffic.

• Proxy302 console — Rotating Residential Proxies
• Proxy302 pricing

Which Proxy302 proxy to buy

Scroll horizontally with the arrow keys to read all content.

Prices in US dollars. Top up your wallet once and pay any line from it. Nothing renews automatically — renew in My assets to keep an IP.

• Proxy302 pricing

Set up scraping proxies in 5 steps

1. Open the setup and pick the line

   The button below opens Rotating Residential Proxies with the AI and automation use case. Under Line choose Budget (US$1.50/GB); Premium (US$5.00/GB) adds the ISP (optional) choice.

2. Buy traffic

   Open Add traffic, pick a package or enter a Custom GB (from 1 GB) and click Buy selected package. Buying traffic and generating a proxy are separate actions.

3. Generate the proxy

   Under Connection settings set Exit mode to Global rotation, or to Choose region / sticky IP and pick Region, State and City. Leave Protocol on HTTP and click Review cost and generate.

4. Build the proxy URL

   In My assets, Copy gives host:port:user:password. Your PROXY_URL is http://USER:[email protected]:2222 — URL-encode the user and password if they contain characters such as @ or :.

5. Run the scraper

   Use the code below: it passes the proxy on every request, waits as long as Retry-After asks and keeps a steady pace. In Scrapy, set request.meta['proxy'] to the same URL.

   Expected result Each page returns 200 through the proxy; a 429 is retried after the wait.

Scraping through a rotating residential proxy (Python requests)

Scroll horizontally with the arrow keys to read all content.

# pip install requests
import os
import time
import requests

# Rotating residential, Exit mode "Global rotation": http://USER:[email protected]:2222
PROXY = os.environ["PROXY_URL"]
PROXIES = {"http": PROXY, "https": PROXY}
session = requests.Session()


def get(url, tries=4):
    for attempt in range(tries):
        # pass proxies on every call: Session.proxies loses to HTTP_PROXY / system proxy settings
        r = session.get(url, proxies=PROXIES, timeout=30)
        if r.status_code not in (429, 500, 502, 503, 504):
            return r
        retry_after = r.headers.get("Retry-After", "")  # seconds the site asks you to wait
        time.sleep(int(retry_after) if retry_after.isdigit() else 2 ** attempt)
    return r


for url in ["https://example.com/catalog?page=1", "https://example.com/catalog?page=2"]:
    r = get(url)
    print(url, r.status_code, len(r.content), "bytes")
    time.sleep(1)  # keep a steady pace: rotation spreads load, it doesn't remove rate limits

Checked on 2026-10-18 (requests 2.33.1) against a local proxy that answers 429 with Retry-After first: both pages were retried after the wait and returned 200.

• Local proxy fixture run
• Requests docs — Proxies (Session.proxies vs environment)

Quick fixes

Web scraping proxy FAQ

Related setups

Price monitoring proxies

Proxies for price monitoring: see competitor prices, promotions and shipping as shoppers in each country and city see them. Rotating residential from US$1.50/GB.

View options and pricing

SERP proxies

SERP proxies for rank tracking: check Google results from the searcher's country and city on household IPs, validate every page and record captchas as unknown.

View options and pricing

Residential proxies for your everyday tools

Real household IPs in 195+ countries with state and city targeting. Rotate on every request or keep sticky sessions. From US$1.50/GB, US$1.35/GB at 100 GB. HTTP and SOCKS5.

View options and pricing