Skip to content

Updated

Proxy for AI Data Collection and RAG:
IP Type, Sessions

  1. For AI data collection and RAG ingestion, use a rotating residential proxy (US$1.50 per GB on Budget) when the pages are country-specific or sensitive to datacenter IPs, and consider the US-only rotating datacenter line (US$0.40 per GB) for tolerant sites and APIs.
  2. Whatever the IP type, crawl respectfully: name your crawler, follow robots.txt (RFC 9309 says it is not an authorization, so check terms and licences too), pace requests and back off on 429.
  3. If the data exists in an open corpus or a licensed feed, start there.

Check it right after buyingExample output

curl -x http://USER:PASS@HOST:PORT https://ipinfo.io/json

ip
203.0.113.24
country
US
Matches the country you picked
Day 15
203.0.113.24
Same address for the whole term
Example addresses (203.0.113.0/24), not real exits
01 / 08Which IP type fits which part of the pipeline

For AI pipelines the proxy is the small part: identity, robots.txt, rate and licence decide what you can collect.

Which IP type fits which part of the pipeline

Rotating residential (195+ countries, HTTP and SOCKS5) fits pages that differ by country or that are strict about the kind of IP. Budget is country level, Premium takes a state or city. Rotating datacenter is US-only with thousands of IPs in rotation at US$0.40 per GB, which suits tolerant sites, documentation and APIs. The datacenter guide lists which sites work with it.

Without a session each new connection gets a new IP, and a reused connection keeps its IP. For multi-page fetches that should look like one visit, use a session (1 to 120 minutes on the Budget residential line, about 10 minutes on rotating datacenter). Traffic is billed per GB, so strip what you do not need: fetch text endpoints, skip images and scripts.

What the published rules say about crawlers

What the published rules say about crawlers
ItemWhat we readWhat it means for your pipeline
RFC 9309 (the robots.txt standard)States that the rules in robots.txt are not a form of access authorization, and that crawlers should not use a cached copy of the file for more than 24 hours unless it cannot be refreshedrobots.txt is the minimum courtesy, not permission; re-read it at least daily and check terms and licences separately
OpenAI crawler pageDescribes separate robots.txt user agents (OAI-SearchBot for search, GPTBot for content that may be used in training) whose settings are independent; disallowing GPTBot indicates a site's content should not be used in trainingSite owners now set rules per purpose (search, training); expect them and respect the purpose a rule names
Common Crawl CCBot pageCCBot names itself in its user agent (CCBot/2.0 with an info URL), can be blocked with User-agent: CCBot and Disallow: / in robots.txt, and runs on dedicated IP ranges with reverse DNS so site owners can verify itA named, verifiable crawler is the norm for large data projects; if the data already exists in an open corpus, using it avoids crawling at all

Sources: RFC 9309, the OpenAI crawler documentation and the Common Crawl CCBot page, opened 20 November 2026. These describe norms and do not replace each site's terms or any licence. This is not legal advice.

A respectful crawl in six habits

Name the crawler in the user agent with a contact address, read robots.txt before each batch and again at least daily, and skip what it closes instead of trying another IP. Honour Crawl-delay where set; otherwise start with a request every several seconds per host and slow down when responses slow down.

Back off on 429 and 503 (use Retry-After when sent), cache what you have fetched so you do not fetch the same page twice, and do not collect personal data or content behind a login. Keep the source URL, fetch time and the licence or terms you relied on next to each document in your index: you will need them when a source asks you to remove something.

A polite fetch loop: robots.txt, Crawl-delay, back-off on 429

import time
import urllib.robotparser
import requests

# One connection copied from My assets (HTTP port 2222; the same login also works over SOCKS5 on 3333).
PROXY = "http://LOGIN:PASSWORD@proxy.proxy302.com:2222"
PROXIES = {"http": PROXY, "https": PROXY}
# A crawler that names itself and says how to reach you is easy for site owners to deal with.
UA = "my-rag-crawler/1.0 (+https://example.com/crawler; contact: you@example.com)"

def polite_get(url, robots, default_delay=5, tries=3):
    if not robots.can_fetch(UA, url):
        return None  # robots.txt says no: skip, do not retry through another IP
    delay = robots.crawl_delay(UA) or default_delay
    for attempt in range(tries):
        r = requests.get(url, proxies=PROXIES, headers={"User-Agent": UA}, timeout=(5, 20))
        if r.status_code != 429:
            time.sleep(delay)
            return r
        wait = int(r.headers.get("Retry-After", "60")) if r.headers.get("Retry-After", "").isdigit() else 60 * (attempt + 1)
        time.sleep(wait)  # back off instead of adding IPs
    return None

site = "https://example.com"
robots = urllib.robotparser.RobotFileParser(site + "/robots.txt")
robots.read()
for path in ["/docs/page-1", "/docs/page-2"]:  # replace with pages you are allowed to read
    r = polite_get(site + path, robots)
    print(path, None if r is None else (r.status_code, len(r.text)))

Illustrative example: replace LOGIN, PASSWORD, the crawler description, the site and the paths. For SOCKS5 install requests[socks] and use socks5h:// with port 3333.

Set it up in 4 steps

  1. Step 1: Choose the IP type

    Open Rotating Residential Proxies for country-specific or strict sites (US$1.50 per GB Budget, US$5 Premium), or Rotating Datacenter (US$0.40 per GB, US only) for tolerant sites and APIs.

  2. Step 2: Generate a connection

    Set the Region (and a session length if one batch should keep one IP) and copy host, port, username and password from My assets.

  3. Step 3: Test the exit

    Open ipinfo.io through the proxy. Expected: the country you set.

  4. Step 4: Crawl politely and log

    Name the crawler, follow robots.txt, pace requests and store source URL, time and terms next to each document.

Quick fixes

Many 429 or 403 responses

Check The request rate is too high or too uniform, or the site blocks that kind of crawler.

Fix Slow down, honour Retry-After and robots.txt; if the site closes the path, skip it.

Pages come back empty for your parser

Check The content is filled in by scripts after load.

Fix Use a browser tool such as Playwright with the same proxy, or look for an official API or feed.

Bandwidth use is higher than expected

Check Images, scripts and repeated fetches count as traffic.

Fix Fetch text only, cache pages and check usage in My assets.

AI data collection FAQ

Which proxy for AI data collection?

Rotating residential for country-specific or strict sites, from US$1.50 per GB; rotating datacenter (US only, US$0.40 per GB) for tolerant sites and APIs. Rate and rules matter more than the IP type.

Does robots.txt give me permission to collect?

No. RFC 9309 says its rules are not a form of access authorization, so check the site terms and any licence as well.

Should my crawler identify itself?

It is the norm for large crawlers: GPTBot and CCBot both name themselves, and site owners set rules per user agent.

Do I need sticky sessions for RAG ingestion?

Only for multi-page flows that should keep one IP; for independent page fetches the default (new IP per new connection) works.

Further reading

Setup guides Proxy302 publishes on GitHub on the same topic:

Reference links

5
  1. RFC 9309: Robots Exclusion Protocol

    Official documentation · Confirmed on:

    Opened on 20 November 2026: States that the rules in robots.txt are not a form of access authorization, and that crawlers should not use a cached copy of the file for more than 24 hours unless it cannot be refreshed

    robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.

  2. OpenAI: Overview of OpenAI crawlers

    Official documentation · Confirmed on:

    Opened on 20 November 2026: Describes separate robots.txt user agents (OAI-SearchBot for search, GPTBot for content that may be used in training) whose settings are independent; disallowing GPTBot indicates a site's content should not be used in training

    robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.

  3. Common Crawl: CCBot

    Official documentation · Confirmed on:

    Opened on 20 November 2026: CCBot names itself in its user agent (CCBot/2.0 with an info URL), can be blocked with User-agent: CCBot and Disallow: / in robots.txt, and runs on dedicated IP ranges with reverse DNS so site owners can verify it

    robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.

  4. Requests documentation: Advanced usage, Proxies

    Official documentation · Confirmed on:

    Opened on 17 November 2026 and re-read 20 November: proxies are passed as a dict to the proxies argument; user:password goes in the URL; SOCKS needs requests[socks]; socks5h resolves DNS on the proxy; requests have no timeout unless one is set.

  5. Proxy302 product facts and price list

    Product observation · Confirmed on:

    Product facts file of the site (dated 10 November 2026), read on 20 November 2026: per-GB prices of both rotating residential lines, traffic does not expire, countries and city targeting, protocols and ports, no free trial.

Ready to try it?

Create an account, top up once, then pick rotating residential, static ISP or datacenter IPs and have one running in your browser within minutes.