HTTP 429 or a consent or check page
Check Requests are too frequent or too uniform.
Fix Wait at least the site's Crawl-delay, cut parallel requests and stop if it persists; adding IPs is not the fix.
Updated
curl -x http://USER:PASS@HOST:PORT https://ipinfo.io/json
Quote pages are light, so the question is pace and rules rather than bandwidth.
Quotes, currency, listing exchange and the language of the page can follow the country the request comes from, and some pages redirect to a regional site. Decide which regional version you need (for example the US site) and set the exit to that country.
Market data is also time-sensitive: a page you read slowly may be seconds or minutes old. Save the request time with every quote, and do not treat scraped quotes as an execution-grade or licensed feed.
| Item | What we read | What it means for a script |
|---|---|---|
| Yahoo Finance (finance.yahoo.com) | A long list of named data and AI crawlers (including GPTBot, ClaudeBot, CCBot and PerplexityBot) is disallowed from the whole site; the group for all other crawlers (*) disallows paths such as /quotes/*, /lookup/, /compare/ and /xhr, and has no crawl-delay line | The site treats named data crawlers strictly; check that your user agent and URLs are not in a closed rule |
| Google Finance (google.com/finance) | The group for all crawlers (*) in google.com/robots.txt has no /finance rule (the only match is an Allow for /m/finance), while /search is disallowed | robots.txt alone does not close /finance pages, but Google's terms of service and automated-traffic checks still apply |
| Nasdaq (nasdaq.com) | All crawlers (*): Crawl-delay 30 (seconds); /api/v1/*, search URLs with q= and the option-chain call/put paths disallowed | One request every 30 seconds is what the file asks for; the internal API paths are closed |
Sources: robots.txt of finance.yahoo.com, google.com and nasdaq.com, and the Nasdaq Data Link front page, opened 20 November 2026. The files change often; read the current one and the site terms before each project. This is not legal advice.
One IP per run is enough for a short batch: a session on the Budget line (1 to 120 minutes) keeps it, and without a session each new connection gets a new IP. Concurrent connections are not limited by the proxy, but honour what the file asks: wait at least the Crawl-delay where one is set, and otherwise start with a request every several seconds.
Do not log in to collect more, do not collect personal data, and stop on 429 or a check page rather than adding IPs. Keep the user agent honest, with a contact address, and store the source URL and request time with every number.
import time
import urllib.robotparser
import requests
# One connection copied from My assets (HTTP port 2222; the same login also works over SOCKS5 on 3333).
PROXY = "http://LOGIN:PASSWORD@proxy.proxy302.com:2222"
PROXIES = {"http": PROXY, "https": PROXY}
UA = "my-market-research/1.0 (contact: you@example.com)"
CHECKS = [
("https://www.nasdaq.com", "/market-activity/stocks/aapl"),
("https://finance.yahoo.com", "/quote/AAPL/"),
] # replace with the public pages you are allowed to read
for site, path in CHECKS:
rp = urllib.robotparser.RobotFileParser(site + "/robots.txt")
rp.read()
if not rp.can_fetch(UA, site + path):
print("robots.txt says no:", site + path)
continue
delay = rp.crawl_delay(UA) or 10 # honour Crawl-delay when the file sets one
r = requests.get(site + path, proxies=PROXIES, headers={"User-Agent": UA}, timeout=(5, 20))
print(site + path, r.status_code, len(r.text), "waiting", delay, "s")
time.sleep(delay)
Illustrative example: replace LOGIN, PASSWORD, the contact address and the pages. For SOCKS5 install requests[socks] and use socks5h:// with port 3333.
Open Rotating Residential Proxies. Budget is US$1.50 per GB (country level); quote pages are small, so a few GB covers many requests.
Set the Region to the country of the site version you read and generate a connection. Copy host, port, username and password from My assets.
Open ipinfo.io through the proxy. Expected: the country you set.
Fetch only paths robots.txt leaves open, wait at least the Crawl-delay, and save the request time with each quote.
Check Requests are too frequent or too uniform.
Fix Wait at least the site's Crawl-delay, cut parallel requests and stop if it persists; adding IPs is not the fix.
Check The exit country or regional site differs from the one you expect.
Fix Check the exit on ipinfo.io and open the regional site you want.
Check The value is filled in by scripts after the page loads.
Fix Use a browser tool such as Playwright with the same proxy, or a licensed data feed.
Rotating residential in the country of the site version you read, from US$1.50 per GB. Yahoo's robots.txt closes /quotes/*, /lookup/ and several other paths and disallows many named data crawlers, so check your URLs first.
Its robots.txt sets Crawl-delay 30 for all crawlers and closes /api/v1/*. Keep to that pace.
We do not advise on trading. Scraped pages can be delayed and incomplete; if you depend on prices, look at a licensed feed such as Nasdaq Data Link and read its terms.
No. A new IP is given on each new connection, and a reused connection keeps its IP. Pace requests rather than chasing IPs.
Setup guides Proxy302 publishes on GitHub on the same topic:
Official documentation · Confirmed on:
Opened on 20 November 2026: A long list of named data and AI crawlers (including GPTBot, ClaudeBot, CCBot and PerplexityBot) is disallowed from the whole site; the group for all other crawlers (*) disallows paths such as /quotes/*, /lookup/, /compare/ and /xhr, and has no crawl-delay line
robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.
Official documentation · Confirmed on:
Opened on 20 November 2026: The group for all crawlers (*) in google.com/robots.txt has no /finance rule (the only match is an Allow for /m/finance), while /search is disallowed
robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.
Official documentation · Confirmed on:
Opened on 20 November 2026: All crawlers (*): Crawl-delay 30 (seconds); /api/v1/*, search URLs with q= and the option-chain call/put paths disallowed
robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.
Third-party source · Confirmed on:
Opened on 20 November 2026: the page is titled "Financial, Economic and Alternative Data | Nasdaq Data Link", a data product separate from the nasdaq.com website; its content is script-rendered and no terms were read.
We only confirmed that the product exists; check its coverage, licence and terms yourself.
Official documentation · Confirmed on:
Opened on 17 November 2026 and re-read 20 November: proxies are passed as a dict to the proxies argument; user:password goes in the URL; SOCKS needs requests[socks]; socks5h resolves DNS on the proxy; requests have no timeout unless one is set.
Product observation · Confirmed on:
Product facts file of the site (dated 10 November 2026), read on 20 November 2026: per-GB prices of both rotating residential lines, traffic does not expire, countries and city targeting, protocols and ports, no free trial.
Create an account, top up once, then pick rotating residential, static ISP or datacenter IPs and have one running in your browser within minutes.