Skip to content

Updated

Proxy for Scraping Indeed and Glassdoor Job Postings

  1. For public job postings on Indeed and Glassdoor, use a rotating residential proxy in the country whose job board you read, and a city if you compare local markets: US$1.50 per GB on Budget (country level), US$5 per GB on Premium (state or city).
  2. Both robots.txt files close parts of the sites, such as Indeed single-posting paths and Glassdoor internal search, so check the exact URLs you request and go slowly.
  3. A proxy changes where requests come from, not what you may collect.

Check it right after buyingExample output

curl -x http://USER:PASS@HOST:PORT https://ipinfo.io/json

ip
203.0.113.24
country
US
Matches the country you picked
Day 15
203.0.113.24
Same address for the whole term
Example addresses (203.0.113.0/24), not real exits
01 / 08Why country and city matter for job postings

Job boards differ by country, so pick the exit country first, then pace slowly and read what robots.txt closes.

Why country and city matter for job postings

Job boards return different listings, salary currencies and languages depending on the country site and the location of the visitor. A search for the same job title from two countries can return two different result sets, so the exit country is part of your data definition.

Set the proxy country to the market you analyse and keep one connection per market. For city-level comparison, the Premium line lets you pick a state or city; Budget is country level only. Rotating residential does not offer city targeting in Russia or Singapore.

What each robots.txt said when we opened it

What each robots.txt said when we opened it
ItemWhat we readWhat it means for a script
Indeed (indeed.com)All crawlers (*): the group opens with Allow: / and then disallows /viewjob, /rc/, /company/*, several /cmp/ review paths (such as /cmp/*/write-review) and /jobs/ browse paths with a country code or title; no crawl-delay lineThe file closes the single-posting page paths; check which exact URLs you request, and keep the pace slow
Glassdoor (glassdoor.com)All crawlers (*): /search/, /jobview/, /ajax/, /profile/ and several /partners/ paths disallowed, plus some /Job/ and /Reviews/ URL patterns; no crawl-delay lineInternal search and ajax paths are off limits for crawlers; read only pages the file leaves open and expect login walls on the rest

Sources: robots.txt of indeed.com and glassdoor.com, opened 20 November 2026. The files differ per country site and change often; read the one for the site you use. This is not legal advice, and site terms may say more.

Session, rate and data rules

Reading one results page then following to the next is a short browsing flow: a session of 10 or 30 minutes on the Budget line keeps one IP across it, while without a session each new connection gets a new IP. Concurrent connections are not limited, but go slowly: one request every several seconds per site is a safe start.

Collect public listing data only. Do not log in to collect more, do not collect candidate or reviewer personal data, and stop on 429 or repeated checks instead of adding IPs. Store the source URL and the collection date with every posting, and remember that postings are removed or edited by employers.

Check robots.txt for your exact URLs, check the exit, fetch slowly

import time
import urllib.robotparser
import requests

# One connection per country, copied from My assets (HTTP port 2222; the same login also works over SOCKS5 on 3333).
PROXY = "http://LOGIN:PASSWORD@proxy.proxy302.com:2222"
PROXIES = {"http": PROXY, "https": PROXY}
UA = "my-jobs-research/1.0 (contact: you@example.com)"
CHECKS = [
    ("https://www.indeed.com", "/jobs?q=data+analyst&l=Austin%2C+TX"),
    ("https://www.glassdoor.com", "/Job/austin-data-analyst-jobs-SRCH_IL.0,6_IC1139761_KO7,19.htm"),
]  # replace with the public pages you are allowed to read

print("exit country:", requests.get("https://ipinfo.io/json", proxies=PROXIES, timeout=(5, 20)).json().get("country"))

for site, path in CHECKS:
    rp = urllib.robotparser.RobotFileParser(site + "/robots.txt")
    rp.read()
    if not rp.can_fetch(UA, site + path):
        print("robots.txt says no:", site + path)
        continue
    r = requests.get(site + path, proxies=PROXIES, headers={"User-Agent": UA}, timeout=(5, 20))
    print(site + path, r.status_code, len(r.text))
    time.sleep(5)  # keep the pace slow; stop on 429 or a check page

Illustrative example: replace LOGIN, PASSWORD, the contact address and the two paths. For SOCKS5 install requests[socks] and use socks5h:// with port 3333.

Set it up in 4 steps

  1. Step 1: Choose the line

    Open Rotating Residential Proxies. Use Budget (US$1.50 per GB, country level) for country-wide job data, or Premium (US$5 per GB) when you need a state or city.

  2. Step 2: One connection per market

    Set the Region to the country (and city on Premium) and generate a connection. Copy host, port, username and password from My assets.

  3. Step 3: Test the exit

    Run the script above or open ipinfo.io through the proxy. Expected: the country you set.

  4. Step 4: Run slowly and save the date

    Fetch only URLs that robots.txt leaves open, pause between requests, and save the posting URL, the market and the date.

Quick fixes

A check page or HTTP 403 appears

Check Requests are too fast or too uniform, or the URL is one the site protects.

Fix Slow down, reduce parallel requests and check the URL against robots.txt; do not rotate IPs to push through.

Results differ from what you see in your browser

Check The exit country or city differs from your browser location.

Fix Check the exit on ipinfo.io and set the same country (or city on Premium).

Python reports a SOCKS proxy error

Check requests[socks] is not installed.

Fix Run python -m pip install "requests[socks]" or use the HTTP line on port 2222.

Job-posting data FAQ

Which proxy for scraping Indeed?

Rotating residential in the country of the Indeed site you read, from US$1.50 per GB. Check robots.txt for your exact URLs first, because single-posting paths are closed there.

Does Glassdoor allow scraping?

Its robots.txt closes /search/, /jobview/ and ajax paths for crawlers, and its terms may say more. Read both before collecting anything.

Do I need city-level IPs?

Only when you compare local markets. Premium lets you pick a state or city; country-wide data works on Budget.

How many GB for a job-data project?

It depends on page size and volume; the GB guide shows how to estimate it.

Further reading

Setup guides Proxy302 publishes on GitHub on the same topic:

Reference links

4
  1. Indeed (indeed.com) robots.txt

    Official documentation · Confirmed on:

    Opened on 20 November 2026: All crawlers (*): the group opens with Allow: / and then disallows /viewjob, /rc/, /company/*, several /cmp/ review paths (such as /cmp/*/write-review) and /jobs/ browse paths with a country code or title; no crawl-delay line

    robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.

  2. Glassdoor (glassdoor.com) robots.txt

    Official documentation · Confirmed on:

    Opened on 20 November 2026: All crawlers (*): /search/, /jobview/, /ajax/, /profile/ and several /partners/ paths disallowed, plus some /Job/ and /Reviews/ URL patterns; no crawl-delay line

    robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.

  3. Requests documentation: Advanced usage, Proxies

    Official documentation · Confirmed on:

    Opened on 17 November 2026 and re-read 20 November: proxies are passed as a dict to the proxies argument; user:password goes in the URL; SOCKS needs requests[socks]; socks5h resolves DNS on the proxy; requests have no timeout unless one is set.

  4. Proxy302 product facts and price list

    Product observation · Confirmed on:

    Product facts file of the site (dated 10 November 2026), read on 20 November 2026: per-GB prices of both rotating residential lines, traffic does not expire, countries and city targeting, protocols and ports, no free trial.

Ready to try it?

Create an account, top up once, then pick rotating residential, static ISP or datacenter IPs and have one running in your browser within minutes.