Skip to content

Updated

Proxy for Scraping Yahoo Finance and Google Finance

  1. Quote pages on Yahoo Finance, Google Finance and Nasdaq are small, so cost is rarely the issue: a rotating residential proxy is US$1.50 per GB on the Budget line.
  2. The rules matter more.
  3. Nasdaq's robots.txt asks for a 30-second crawl delay, Yahoo Finance disallows a long list of named data and AI crawlers, and Google's robots.txt has no /finance rule but its terms and traffic checks still apply. For prices you must depend on, check whether a licensed data feed fits before scraping.

Check it right after buyingExample output

curl -x http://USER:PASS@HOST:PORT https://ipinfo.io/json

ip
203.0.113.24
country
US
Matches the country you picked
Day 15
203.0.113.24
Same address for the whole term
Example addresses (203.0.113.0/24), not real exits
01 / 08What the exit country changes for market data

Quote pages are light, so the question is pace and rules rather than bandwidth.

What the exit country changes for market data

Quotes, currency, listing exchange and the language of the page can follow the country the request comes from, and some pages redirect to a regional site. Decide which regional version you need (for example the US site) and set the exit to that country.

Market data is also time-sensitive: a page you read slowly may be seconds or minutes old. Save the request time with every quote, and do not treat scraped quotes as an execution-grade or licensed feed.

What each robots.txt said when we opened it

What each robots.txt said when we opened it
ItemWhat we readWhat it means for a script
Yahoo Finance (finance.yahoo.com)A long list of named data and AI crawlers (including GPTBot, ClaudeBot, CCBot and PerplexityBot) is disallowed from the whole site; the group for all other crawlers (*) disallows paths such as /quotes/*, /lookup/, /compare/ and /xhr, and has no crawl-delay lineThe site treats named data crawlers strictly; check that your user agent and URLs are not in a closed rule
Google Finance (google.com/finance)The group for all crawlers (*) in google.com/robots.txt has no /finance rule (the only match is an Allow for /m/finance), while /search is disallowedrobots.txt alone does not close /finance pages, but Google's terms of service and automated-traffic checks still apply
Nasdaq (nasdaq.com)All crawlers (*): Crawl-delay 30 (seconds); /api/v1/*, search URLs with q= and the option-chain call/put paths disallowedOne request every 30 seconds is what the file asks for; the internal API paths are closed

Sources: robots.txt of finance.yahoo.com, google.com and nasdaq.com, and the Nasdaq Data Link front page, opened 20 November 2026. The files change often; read the current one and the site terms before each project. This is not legal advice.

Sessions, rate and what to avoid

One IP per run is enough for a short batch: a session on the Budget line (1 to 120 minutes) keeps it, and without a session each new connection gets a new IP. Concurrent connections are not limited by the proxy, but honour what the file asks: wait at least the Crawl-delay where one is set, and otherwise start with a request every several seconds.

Do not log in to collect more, do not collect personal data, and stop on 429 or a check page rather than adding IPs. Keep the user agent honest, with a contact address, and store the source URL and request time with every number.

Check robots.txt, honour Crawl-delay, fetch one quote page

import time
import urllib.robotparser
import requests

# One connection copied from My assets (HTTP port 2222; the same login also works over SOCKS5 on 3333).
PROXY = "http://LOGIN:PASSWORD@proxy.proxy302.com:2222"
PROXIES = {"http": PROXY, "https": PROXY}
UA = "my-market-research/1.0 (contact: you@example.com)"
CHECKS = [
    ("https://www.nasdaq.com", "/market-activity/stocks/aapl"),
    ("https://finance.yahoo.com", "/quote/AAPL/"),
]  # replace with the public pages you are allowed to read

for site, path in CHECKS:
    rp = urllib.robotparser.RobotFileParser(site + "/robots.txt")
    rp.read()
    if not rp.can_fetch(UA, site + path):
        print("robots.txt says no:", site + path)
        continue
    delay = rp.crawl_delay(UA) or 10  # honour Crawl-delay when the file sets one
    r = requests.get(site + path, proxies=PROXIES, headers={"User-Agent": UA}, timeout=(5, 20))
    print(site + path, r.status_code, len(r.text), "waiting", delay, "s")
    time.sleep(delay)

Illustrative example: replace LOGIN, PASSWORD, the contact address and the pages. For SOCKS5 install requests[socks] and use socks5h:// with port 3333.

Set it up in 4 steps

  1. Step 1: Choose the line

    Open Rotating Residential Proxies. Budget is US$1.50 per GB (country level); quote pages are small, so a few GB covers many requests.

  2. Step 2: Set the country

    Set the Region to the country of the site version you read and generate a connection. Copy host, port, username and password from My assets.

  3. Step 3: Test the exit

    Open ipinfo.io through the proxy. Expected: the country you set.

  4. Step 4: Run at the file's pace

    Fetch only paths robots.txt leaves open, wait at least the Crawl-delay, and save the request time with each quote.

Quick fixes

HTTP 429 or a consent or check page

Check Requests are too frequent or too uniform.

Fix Wait at least the site's Crawl-delay, cut parallel requests and stop if it persists; adding IPs is not the fix.

Quotes show another currency or exchange

Check The exit country or regional site differs from the one you expect.

Fix Check the exit on ipinfo.io and open the regional site you want.

The page loads but the number is missing

Check The value is filled in by scripts after the page loads.

Fix Use a browser tool such as Playwright with the same proxy, or a licensed data feed.

Financial data FAQ

Which proxy for scraping Yahoo Finance?

Rotating residential in the country of the site version you read, from US$1.50 per GB. Yahoo's robots.txt closes /quotes/*, /lookup/ and several other paths and disallows many named data crawlers, so check your URLs first.

Does Nasdaq limit crawlers?

Its robots.txt sets Crawl-delay 30 for all crawlers and closes /api/v1/*. Keep to that pace.

Is scraped market data good enough for trading?

We do not advise on trading. Scraped pages can be delayed and incomplete; if you depend on prices, look at a licensed feed such as Nasdaq Data Link and read its terms.

Do I need a new IP for every request?

No. A new IP is given on each new connection, and a reused connection keeps its IP. Pace requests rather than chasing IPs.

Further reading

Setup guides Proxy302 publishes on GitHub on the same topic:

Reference links

6
  1. Yahoo Finance (finance.yahoo.com) robots.txt

    Official documentation · Confirmed on:

    Opened on 20 November 2026: A long list of named data and AI crawlers (including GPTBot, ClaudeBot, CCBot and PerplexityBot) is disallowed from the whole site; the group for all other crawlers (*) disallows paths such as /quotes/*, /lookup/, /compare/ and /xhr, and has no crawl-delay line

    robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.

  2. Google Finance (google.com/finance) robots.txt

    Official documentation · Confirmed on:

    Opened on 20 November 2026: The group for all crawlers (*) in google.com/robots.txt has no /finance rule (the only match is an Allow for /m/finance), while /search is disallowed

    robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.

  3. Nasdaq (nasdaq.com) robots.txt

    Official documentation · Confirmed on:

    Opened on 20 November 2026: All crawlers (*): Crawl-delay 30 (seconds); /api/v1/*, search URLs with q= and the option-chain call/put paths disallowed

    robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.

  4. Nasdaq Data Link

    Third-party source · Confirmed on:

    Opened on 20 November 2026: the page is titled "Financial, Economic and Alternative Data | Nasdaq Data Link", a data product separate from the nasdaq.com website; its content is script-rendered and no terms were read.

    We only confirmed that the product exists; check its coverage, licence and terms yourself.

  5. Requests documentation: Advanced usage, Proxies

    Official documentation · Confirmed on:

    Opened on 17 November 2026 and re-read 20 November: proxies are passed as a dict to the proxies argument; user:password goes in the URL; SOCKS needs requests[socks]; socks5h resolves DNS on the proxy; requests have no timeout unless one is set.

  6. Proxy302 product facts and price list

    Product observation · Confirmed on:

    Product facts file of the site (dated 10 November 2026), read on 20 November 2026: per-GB prices of both rotating residential lines, traffic does not expire, countries and city targeting, protocols and ports, no free trial.

Ready to try it?

Create an account, top up once, then pick rotating residential, static ISP or datacenter IPs and have one running in your browser within minutes.