Many 429 or 403 responses
Check The request rate is too high or too uniform, or the site blocks that kind of crawler.
Fix Slow down, honour Retry-After and robots.txt; if the site closes the path, skip it.
Updated
curl -x http://USER:PASS@HOST:PORT https://ipinfo.io/json
For AI pipelines the proxy is the small part: identity, robots.txt, rate and licence decide what you can collect.
Rotating residential (195+ countries, HTTP and SOCKS5) fits pages that differ by country or that are strict about the kind of IP. Budget is country level, Premium takes a state or city. Rotating datacenter is US-only with thousands of IPs in rotation at US$0.40 per GB, which suits tolerant sites, documentation and APIs. The datacenter guide lists which sites work with it.
Without a session each new connection gets a new IP, and a reused connection keeps its IP. For multi-page fetches that should look like one visit, use a session (1 to 120 minutes on the Budget residential line, about 10 minutes on rotating datacenter). Traffic is billed per GB, so strip what you do not need: fetch text endpoints, skip images and scripts.
| Item | What we read | What it means for your pipeline |
|---|---|---|
| RFC 9309 (the robots.txt standard) | States that the rules in robots.txt are not a form of access authorization, and that crawlers should not use a cached copy of the file for more than 24 hours unless it cannot be refreshed | robots.txt is the minimum courtesy, not permission; re-read it at least daily and check terms and licences separately |
| OpenAI crawler page | Describes separate robots.txt user agents (OAI-SearchBot for search, GPTBot for content that may be used in training) whose settings are independent; disallowing GPTBot indicates a site's content should not be used in training | Site owners now set rules per purpose (search, training); expect them and respect the purpose a rule names |
| Common Crawl CCBot page | CCBot names itself in its user agent (CCBot/2.0 with an info URL), can be blocked with User-agent: CCBot and Disallow: / in robots.txt, and runs on dedicated IP ranges with reverse DNS so site owners can verify it | A named, verifiable crawler is the norm for large data projects; if the data already exists in an open corpus, using it avoids crawling at all |
Sources: RFC 9309, the OpenAI crawler documentation and the Common Crawl CCBot page, opened 20 November 2026. These describe norms and do not replace each site's terms or any licence. This is not legal advice.
Name the crawler in the user agent with a contact address, read robots.txt before each batch and again at least daily, and skip what it closes instead of trying another IP. Honour Crawl-delay where set; otherwise start with a request every several seconds per host and slow down when responses slow down.
Back off on 429 and 503 (use Retry-After when sent), cache what you have fetched so you do not fetch the same page twice, and do not collect personal data or content behind a login. Keep the source URL, fetch time and the licence or terms you relied on next to each document in your index: you will need them when a source asks you to remove something.
import time
import urllib.robotparser
import requests
# One connection copied from My assets (HTTP port 2222; the same login also works over SOCKS5 on 3333).
PROXY = "http://LOGIN:PASSWORD@proxy.proxy302.com:2222"
PROXIES = {"http": PROXY, "https": PROXY}
# A crawler that names itself and says how to reach you is easy for site owners to deal with.
UA = "my-rag-crawler/1.0 (+https://example.com/crawler; contact: you@example.com)"
def polite_get(url, robots, default_delay=5, tries=3):
if not robots.can_fetch(UA, url):
return None # robots.txt says no: skip, do not retry through another IP
delay = robots.crawl_delay(UA) or default_delay
for attempt in range(tries):
r = requests.get(url, proxies=PROXIES, headers={"User-Agent": UA}, timeout=(5, 20))
if r.status_code != 429:
time.sleep(delay)
return r
wait = int(r.headers.get("Retry-After", "60")) if r.headers.get("Retry-After", "").isdigit() else 60 * (attempt + 1)
time.sleep(wait) # back off instead of adding IPs
return None
site = "https://example.com"
robots = urllib.robotparser.RobotFileParser(site + "/robots.txt")
robots.read()
for path in ["/docs/page-1", "/docs/page-2"]: # replace with pages you are allowed to read
r = polite_get(site + path, robots)
print(path, None if r is None else (r.status_code, len(r.text)))
Illustrative example: replace LOGIN, PASSWORD, the crawler description, the site and the paths. For SOCKS5 install requests[socks] and use socks5h:// with port 3333.
Open Rotating Residential Proxies for country-specific or strict sites (US$1.50 per GB Budget, US$5 Premium), or Rotating Datacenter (US$0.40 per GB, US only) for tolerant sites and APIs.
Set the Region (and a session length if one batch should keep one IP) and copy host, port, username and password from My assets.
Open ipinfo.io through the proxy. Expected: the country you set.
Name the crawler, follow robots.txt, pace requests and store source URL, time and terms next to each document.
Check The request rate is too high or too uniform, or the site blocks that kind of crawler.
Fix Slow down, honour Retry-After and robots.txt; if the site closes the path, skip it.
Check The content is filled in by scripts after load.
Fix Use a browser tool such as Playwright with the same proxy, or look for an official API or feed.
Check Images, scripts and repeated fetches count as traffic.
Fix Fetch text only, cache pages and check usage in My assets.
Rotating residential for country-specific or strict sites, from US$1.50 per GB; rotating datacenter (US only, US$0.40 per GB) for tolerant sites and APIs. Rate and rules matter more than the IP type.
No. RFC 9309 says its rules are not a form of access authorization, so check the site terms and any licence as well.
It is the norm for large crawlers: GPTBot and CCBot both name themselves, and site owners set rules per user agent.
Only for multi-page flows that should keep one IP; for independent page fetches the default (new IP per new connection) works.
Setup guides Proxy302 publishes on GitHub on the same topic:
Official documentation · Confirmed on:
Opened on 20 November 2026: States that the rules in robots.txt are not a form of access authorization, and that crawlers should not use a cached copy of the file for more than 24 hours unless it cannot be refreshed
robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.
Official documentation · Confirmed on:
Opened on 20 November 2026: Describes separate robots.txt user agents (OAI-SearchBot for search, GPTBot for content that may be used in training) whose settings are independent; disallowing GPTBot indicates a site's content should not be used in training
robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.
Official documentation · Confirmed on:
Opened on 20 November 2026: CCBot names itself in its user agent (CCBot/2.0 with an info URL), can be blocked with User-agent: CCBot and Disallow: / in robots.txt, and runs on dedicated IP ranges with reverse DNS so site owners can verify it
robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.
Official documentation · Confirmed on:
Opened on 17 November 2026 and re-read 20 November: proxies are passed as a dict to the proxies argument; user:password goes in the URL; SOCKS needs requests[socks]; socks5h resolves DNS on the proxy; requests have no timeout unless one is set.
Product observation · Confirmed on:
Product facts file of the site (dated 10 November 2026), read on 20 November 2026: per-GB prices of both rotating residential lines, traffic does not expire, countries and city targeting, protocols and ports, no free trial.
Create an account, top up once, then pick rotating residential, static ISP or datacenter IPs and have one running in your browser within minutes.