Scrapy Rotating Proxy: Middleware, Retries, Status Codes

One endpoint, two ways to set it

The rotating proxy is one HOST and PORT with a username and password; the exit IP changes on the server side, so you do not manage an IP list. In Scrapy the proxy is read from the request meta key proxy, and credentials can sit in the URL as http://username:password@some_proxy_server:port. The meta value takes precedence over the http_proxy and https_proxy environment variables.

Set it on each Request when only some requests need the proxy, or in a small downloader middleware when all of them do. The existing guide to Scrapy proxy setup covers environment variables and special characters in passwords; this page is about rotation and retries.

• Scrapy proxy setup
• Python requests proxy setup

• Sources:
• Scrapy docs: Downloader Middleware
• Proxy302 product facts and price list

Downloader middleware that sets the proxy

# middlewares.py
class ProxyMiddleware:
    def process_request(self, request, spider):
        # Only set it when the request does not carry its own proxy.
        request.meta.setdefault("proxy", "http://LOGIN:PASSWORD@HOST:PORT")

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyMiddleware": 350,
}

Example only. Priority 350 runs before the built-in HttpProxyMiddleware, which reads the meta value.

• Sources:
• Scrapy docs: Downloader Middleware

A rotating IP is not one IP per request

A rotating endpoint gives a new IP for each new connection. Scrapy may keep a connection open and send several requests over it, so several requests can share one IP. Do not assume one IP per request: log the exit IP as in the spider below, then decide.

When a task must stay on one address, for example a login followed by a few pages, generate the proxy in the sticky mode ("Choose region / sticky IP") and keep those requests together. For independent pages, use the "Choose region / new IP per connection" mode.

• rotating vs sticky sessions

• Sources:
• Proxy302 product facts and price list

A spider that prints the exit IP

import scrapy

PROXY = "http://LOGIN:PASSWORD@HOST:PORT"

class IpCheckSpider(scrapy.Spider):
    name = "ip_check"

    def start_requests(self):
        for i in range(5):
            yield scrapy.Request(
                "https://ipinfo.io/ip?n=%d" % i,
                meta={"proxy": PROXY, "max_retry_times": 3},
                dont_filter=True,
            )

    def parse(self, response):
        self.logger.info("exit IP: %s", response.text.strip())

Example only. Fill in the proxy from My assets, run it with scrapy crawl ip_check and read the logged IPs. start_requests is still the classic entry point; newer Scrapy versions also offer an async start().

• Sources:
• Scrapy docs: Downloader Middleware

Which status codes to retry

Source: Scrapy downloader middleware documentation (default RETRY_HTTP_CODES), read 12 November 2026; the "what to do" column is a recommendation. Defaults can differ between Scrapy versions.

• Sources:
• Scrapy docs: Downloader Middleware
• MDN: 429 Too Many Requests

Retry and politeness settings

# settings.py
ROBOTSTXT_OBEY = True
DOWNLOAD_TIMEOUT = 30

# retries (the documented defaults)
RETRY_ENABLED = True
RETRY_TIMES = 2                      # extra tries after the first download
RETRY_HTTP_CODES = [408, 429, 500, 502, 503, 504, 522, 524]

# be gentle with the target site
DOWNLOAD_DELAY = 1.0
CONCURRENT_REQUESTS_PER_DOMAIN = 4
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0

Example only. RETRY_TIMES counts extra tries after the first download. Per request you can set meta["max_retry_times"], or meta["dont_retry"] = True to skip retries.

• Sources:
• Scrapy docs: Downloader Middleware
• Scrapy docs: AutoThrottle extension
• Scrapy docs: Settings

AutoThrottle and delays

AutoThrottle is off by default. When enabled it sets the delay from how fast the site answers, never goes below DOWNLOAD_DELAY, and responses other than 200 cannot shorten it, so errors slow the crawl down instead of speeding it up. START_DELAY is 5.0, MAX_DELAY 60.0 and TARGET_CONCURRENCY 1.0 by default.

Keep ROBOTSTXT_OBEY enabled, set DOWNLOAD_DELAY and CONCURRENT_REQUESTS_PER_DOMAIN explicitly instead of relying on defaults, and read the site terms. A proxy changes where requests come from; it does not change how many requests the site can take.

• Sources:
• Scrapy docs: AutoThrottle extension
• Scrapy docs: Settings

Set it up in four steps

1. Step 1: Generate the proxy

   On the rotating residential page choose the country, the exit mode and HTTP or SOCKS5, then generate. HTTP uses port 2222 and SOCKS5 uses port 3333.

   • rotating residential page

2. Step 2: Add the proxy

   Copy HOST, PORT, LOGIN and PASSWORD from My assets and set meta["proxy"] or add the middleware.

   • Scrapy proxy setup

3. Step 3: Check the exit IPs

   Run the ip_check spider and read the five logged IPs.

   Expected result Five IPs, with the country you chose.

4. Step 4: Add the retry and delay settings

   Paste the settings, run a small crawl and look at the retry and 429 counts in the Scrapy stats at the end.

Troubleshooting

FAQ

Read next

Scrapy proxy setup

Use an HTTP proxy in Scrapy with request meta, a downloader middleware or environment variables, and check the exit IP.

Python requests proxy setup

How to use HTTP and SOCKS5 proxies in Python requests: examples, timeouts and an IP check.

Rotating vs sticky sessions

Three Exit mode options decide whether a residential proxy keeps one IP or changes it; new proxies start on sticky.

Rotating datacenter proxies

US datacenter IPs billed per GB from US$0.40/GB, with a new IP on each new connection from a large rotating pool. HTTP and SOCKS5, no packs to buy.

View options and pricing