Scrapy Proxy Setup: Request Meta, Middleware and Environment Variables

The proxy meta key

The Scrapy documentation describes HttpProxyMiddleware as the component that sets the HTTP proxy for requests through the proxy meta value of a Request object. You can set that key yourself, to a value like http://some_proxy_server:port or http://username:password@some_proxy_server:port. The docs add that the per-request value takes precedence over the environment variables and that it ignores no_proxy.

The proxy meta key: example

import scrapy

class IpSpider(scrapy.Spider):
    name = "ip"

    async def start(self):
        yield scrapy.Request(
            "https://ipinfo.io/json",
            meta={"proxy": "http://USERNAME:PASSWORD@proxy-host:port"},
        )

    def parse(self, response):
        yield response.json()

Replace USERNAME, PASSWORD, proxy-host and port with the values of your proxy.

• Sources:
• Scrapy documentation: Downloader Middleware, HttpProxyMiddleware and RetryMiddleware
• Scrapy documentation: Requests and Responses

One proxy for every request

To avoid repeating the meta key, set it in a downloader middleware. The documentation says a downloader middleware is activated through the DOWNLOADER_MIDDLEWARES setting, where each class path has an order number, and that process_request() is called for each request that goes through it.

One proxy for every request: example

# myproject/middlewares.py
class ProxyMiddleware:
    def process_request(self, request):
        request.meta.setdefault("proxy", "http://USERNAME:PASSWORD@proxy-host:port")
        return None

Example only: replace the placeholder values with your own.

One proxy for every request: example

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyMiddleware": 543,
}

Read the credentials from an environment variable or a secrets file instead of writing them in the code. Check the process_request() signature in the documentation of the Scrapy version you run.

• Sources:
• Scrapy documentation: Downloader Middleware, HttpProxyMiddleware and RetryMiddleware

Environment variables

HttpProxyMiddleware obeys, like Python's urllib.request, the http_proxy, https_proxy and no_proxy environment variables. Set them before you start the crawl and you do not need to change any spider code.

Environment variables: example

export http_proxy="http://USERNAME:PASSWORD@proxy-host:port"
export https_proxy="http://USERNAME:PASSWORD@proxy-host:port"
scrapy crawl ip

Special characters such as @ and : in a password must be percent-encoded in a proxy URL. See the [proxy address, port and login guide](/guides/proxy-address-port-login/).

• Sources:
• Scrapy documentation: Downloader Middleware, HttpProxyMiddleware and RetryMiddleware

HTTP or SOCKS

The Scrapy documentation says that a SOCKS proxy URL such as socks5://username:password@some_proxy_server:port can be used only if the download handler supports it, and it names the httpx-based handler as the built-in one that does, while the other built-in handlers do not. The same page notes that proxy URLs usually use the http:// scheme. For a default Scrapy project, use an HTTP proxy. If you need SOCKS5, check which download handler you use and read the handler's documentation first.

• Sources:
• Scrapy documentation: Downloader Middleware, HttpProxyMiddleware and RetryMiddleware

Check that the proxy is used

Run the spider above and read the country and org fields in the response from https://ipinfo.io/json; the proxy location guide explains how to read them. If you see your own address, the meta key was not set on that request, or the environment variables were set after the crawl started. Add the middleware so every request has the key.

• proxy location guide

Keep the crawl considerate

A proxy changes the address a site sees, but the site's rules still apply. Scrapy has settings for this. The settings documentation says ROBOTSTXT_OBEY makes Scrapy respect robots.txt policies, and that it is enabled in the settings.py generated by the scrapy startproject command. DOWNLOAD_DELAY sets the minimum seconds between two consecutive requests to the same domain and is meant to avoid hitting servers too hard, and CONCURRENT_REQUESTS_PER_DOMAIN limits simultaneous requests to a domain. The AutoThrottle extension adjusts delays dynamically, using download latency, to reach a target concurrency per remote site. Read the target site's terms before you crawl it.

Keep the crawl considerate: example

# settings.py
ROBOTSTXT_OBEY = True
DOWNLOAD_DELAY = 1
CONCURRENT_REQUESTS_PER_DOMAIN = 2
AUTOTHROTTLE_ENABLED = True

These values are a starting point; choose numbers that suit the site and your task.

• Sources:
• Scrapy documentation: Settings
• Scrapy documentation: AutoThrottle extension

Common errors

A 407 status means the proxy did not accept the username and password: copy them again and check the encoding, then see the authentication required guide. Connection errors and timeouts usually mean a wrong host, port or scheme; the connection failed guide lists the checks. The documentation describes RetryMiddleware as retrying requests that fail for temporary reasons such as a connection timeout or an HTTP 500 error, so a few failures in the log are not always a proxy problem. If you collect pages with plain HTTP calls instead of Scrapy, see the Python requests guide.

• authentication required guide
• connection failed guide
• Python requests guide

• Sources:
• Scrapy documentation: Downloader Middleware, HttpProxyMiddleware and RetryMiddleware

How to do it with Proxy302

Any of these setups only needs the host, port, username and password of a Proxy302 proxy. Rotating residential traffic is billed per GB and the exit country is set when you generate the proxy; HTTP is on port 2222 of proxy.proxy302.com. A static ISP IP stays fixed for the whole lease term and does not rotate. The table shows exits in 7 countries from our latest hourly check; the probe sends its request to ChatGPT, which is only an example target. Details are on the rotating residential page and in the proxy location guide.

• rotating residential page
• proxy location guide

Measured exits: ChatGPT from 7 countries

Hourly check of the Proxy302 probe, 28 October 2026, 14:56 UTC: one request to ChatGPT per country through a dynamic residential exit, using the public Proxy302 entry. Only the first three octets of an IP are public. One sample per country, not an average; later checks will differ.

• Sources:
• Proxy302 hourly connectivity check

How to do it with Proxy302: steps

1. Step 1

   Open the rotating residential page, choose the line, and keep Protocol on HTTP or choose SOCKS5, then click Review cost and generate.

2. Step 2

   Open My assets (the console page that holds everything you bought) and copy HOST, PORT, USERNAME and PASSWORD. The Connect page shows a ready example with your own endpoint.

3. Step 3

   Enter the details in the browser or tool you are setting up.

4. Step 4

   Confirm the exit with https://ipinfo.io/json through the proxy.

5. Step 5

   From mainland China, start the helper tool from the Connect page first and keep it open while you use the proxy.

   • Helper tool guide

Prices and limits

The Budget line costs US$1.50 per GB, US$1.35 per GB in a 100 GB package and US$1.20 per GB at 300 GB. There is no free trial, and the smallest purchase is 1 GB. Which exit locations are available changes daily.

• Sources:
• Proxy302 product facts and price list

FAQ

Read next

Proxy address, port, username and password

Every working proxy is described by five basic fields: protocol, host address, port, login, and password.

407 Proxy Authentication Required

HTTP 407 Proxy Authentication Required comes from the proxy server, not from the website you are trying to reach.

Fix ERR_PROXY_CONNECTION_FAILED

The ERR_PROXY_CONNECTION_FAILED error appears in Chrome and Edge when the browser cannot establish a connection with a proxy server.

Check a proxy's IP, country and ISP

Check the exit IP, country and ISP your proxy really gives you in 30 seconds — and why IP databases disagree on the city. With our own test of three lookup services.

Rotating residential proxies

Real household IPs in 195+ countries with state and city targeting. Rotate on every request or keep sticky sessions. From US$1.50/GB, US$1.35/GB at 100 GB. HTTP and SOCKS5.

View options and pricing

Python requests proxy setup

How to use HTTP and SOCKS5 proxies in Python requests: examples, timeouts and an IP check.