Proxies for Web Scraping: How to Choose the Type, Rotation and Checks

What web scraping is and when you need a proxy

Web scraping is the automatic collection of data from a site's pages. You need a proxy when requests must go out from a different address: for example, to see a page the way a user in another country sees it. If the site has an official API with the data you need, it is often simpler and more reliable to use it: you may not need a proxy at all.

The site's rules come first

A proxy changes only the address the site sees. The site's rules still apply. Read the terms of use and the robots.txt file. The RFC 9309 standard describes robots.txt as a set of rules that crawlers are asked to follow; the file must sit at the path /robots.txt. The standard states plainly that these rules are not a form of access authorization. It also describes what to do on errors: if the file is unavailable with a 4xx code, the crawler may access any resources, and if it is unavailable because of a server error (5xx), the crawler must assume that access is completely closed.

• Sources:
• RFC 9309: Robots Exclusion Protocol

Proxy types for collecting data

• Rotating residential proxies: the address changes, and billing is usually per GB. They suit a large number of independent requests to different pages. • Static ISP: one address is fixed for the whole lease term and does not rotate. They suit cases where you need the same address for the term. Static ISP IPs leased for 30 days or longer are dedicated. • Datacenter proxies: leased per IP for a term, and the address belongs to a datacenter.

The differences between them are covered in detail in the comparison of residential, ISP and datacenter proxies, so we do not repeat them here. For more on rotating proxies, see the page on rotating residential proxies; for static ones, see the page on static residential proxies.

• comparison of residential, ISP and datacenter proxies
• rotating residential proxies
• static residential proxies

Rotation or a fixed address

Rotation spreads requests across many addresses. A fixed address is handy when you need to keep a session on one address, for example when you log in to an account. Choose by the task: for occasional independent requests, rotation fits; for work within a single session, a fixed address fits.

HTTP or SOCKS5

Most web scraping libraries work with HTTP proxies equally well. You need SOCKS5 if your program or library supports exactly that. Check the documentation of your tool: for example, in Scrapy, SOCKS support depends on the download handler, see the guide to proxies in Scrapy.

• guide to proxies in Scrapy

Selection checklist

1. Step 1

   Billing unit: per GB, or per IP with a lease for a term.

2. Step 2

   Minimum order: for example, 1 GB of traffic or one IP for one day.

3. Step 3

   Geotargeting: countries and, if needed, cities.

4. Step 4

   Protocol: HTTP or SOCKS5, and how it is selected.

5. Step 5

   Check before launch: buy the minimum, send one request through the proxy to https://ipinfo.io/json and check the country and org fields.

How to check the proxy and set it in code

Open https://ipinfo.io/json through the proxy and make sure the country and network match what you expect; how to read the fields is explained in the guide to checking a proxy's location. The address, port, username and password are described in the guide to proxy details.

The Requests documentation says that a proxy is passed in the proxies argument, and advises setting a timeout for most requests to external servers: without it, the code can hang for a long time. More detail is in the requests guide.

• guide to checking a proxy's location
• guide to proxy details
• requests guide

How to check the proxy and set it in code: example

import requests

proxy = "http://USERNAME:PASSWORD@proxy-host:port"
proxies = {"http": proxy, "https": proxy}

r = requests.get("https://ipinfo.io/json", proxies=proxies, timeout=(5, 20))
print(r.json())

Replace USERNAME, PASSWORD, proxy-host and port with your proxy's details.

• Sources:
• Requests documentation: Advanced Usage (proxies and timeouts)

A gentle load on the site

A scraper should not overload the site. In Scrapy, the DOWNLOAD_DELAY setting sets the minimum pause between two requests to the same domain, and the documentation explains that it is there so as not to load servers too heavily. The AutoThrottle extension picks pauses dynamically based on the site's response time. The ROBOTSTXT_OBEY parameter makes Scrapy follow robots.txt. The same principles apply in other tools: a pause between requests, a limit on concurrency and timeouts.

• Sources:
• Scrapy documentation: Settings
• Scrapy documentation: AutoThrottle extension

Common errors

Code 407 means the proxy did not accept the username and password: copy the details again and check the encoding of special characters, see the guide to error 407. A timeout usually points to a wrong port or protocol type; the checks are listed in the guide to connection failures.

• guide to error 407
• guide to connection failures

How to do it with Proxy302

Any of these tools needs only the address, port, username and password of a Proxy302 proxy. Rotating residential proxy traffic is billed per GB, and the exit country is set when you create the proxy; HTTP is port 2222 and SOCKS5 is port 3333 on proxy.proxy302.com. A static ISP IP is fixed for the whole lease term and does not rotate. The table shows exits in 7 countries at our latest hourly check; the probe sends a request to ChatGPT, which is only an example of a target site. For details, see the rotating residential proxies page and the guide to checking a proxy's location.

• rotating residential proxies page
• guide to checking a proxy's location

Measured exits: ChatGPT from 7 countries

Hourly check of the Proxy302 probe, 28 October 2026, 14:56 UTC: one request to ChatGPT per country through a dynamic residential exit, using the public Proxy302 entry. Only the first three octets of an IP are public. One sample per country, not an average; later checks will differ.

• Sources:
• Proxy302 hourly connectivity check

How to do it with Proxy302: steps

1. Step 1

   Open the rotating residential page, choose the line, and keep Protocol on HTTP or choose SOCKS5, then click Review cost and generate.

2. Step 2

   Open My assets (the console page that holds everything you bought) and copy HOST, PORT, USERNAME and PASSWORD. The Connect page shows a ready example with your own endpoint.

3. Step 3

   Enter the details in the browser or tool you are setting up.

4. Step 4

   Confirm the exit with https://ipinfo.io/json through the proxy.

5. Step 5

   From mainland China, start the helper tool from the Connect page first and keep it open while you use the proxy.

   • Helper tool guide

Prices and limits

The Budget line costs US$1.50 per GB, US$1.35 per GB in a 100 GB package and US$1.20 per GB at 300 GB. There is no free trial, and the smallest purchase is 1 GB. Which exit locations are available changes daily.

• Sources:
• Proxy302 product facts and price list

FAQ

Read next

Proxy address, port, username and password

Every working proxy is described by five basic fields: protocol, host address, port, login, and password.

Check a proxy's IP, country and ISP

Check the exit IP, country and ISP your proxy really gives you in 30 seconds — and why IP databases disagree on the city. With our own test of three lookup services.

Rotating residential proxies

Real household IPs in 195+ countries with state and city targeting. Rotate on every request or keep sticky sessions. From US$1.50/GB, US$1.35/GB at 100 GB. HTTP and SOCKS5.

View options and pricing

ISP proxies (static residential)

Fixed static ISP proxies: a residential-grade IP that stays the same for the whole term. For social accounts, stores and AI tools. HTTP and SOCKS5.

View options and pricing

Python requests proxy setup

How to use HTTP and SOCKS5 proxies in Python requests: examples, timeouts and an IP check.

Scrapy proxy setup

Use an HTTP proxy in Scrapy with request meta, a downloader middleware or environment variables, and check the exit IP.

Residential vs ISP vs datacenter proxies

What separates rotating residential, static ISP and datacenter proxies — who owns the IP, whether it changes, how you pay — and which one Proxy302's own presets pick for each job.