A check page or HTTP 403 appears
Check Requests are too fast or too uniform, or the URL is one the site protects.
Fix Slow down, reduce parallel requests and check the URL against robots.txt; do not rotate IPs to push through.
Updated
curl -x http://USER:PASS@HOST:PORT https://ipinfo.io/json
Job boards differ by country, so pick the exit country first, then pace slowly and read what robots.txt closes.
Job boards return different listings, salary currencies and languages depending on the country site and the location of the visitor. A search for the same job title from two countries can return two different result sets, so the exit country is part of your data definition.
Set the proxy country to the market you analyse and keep one connection per market. For city-level comparison, the Premium line lets you pick a state or city; Budget is country level only. Rotating residential does not offer city targeting in Russia or Singapore.
| Item | What we read | What it means for a script |
|---|---|---|
| Indeed (indeed.com) | All crawlers (*): the group opens with Allow: / and then disallows /viewjob, /rc/, /company/*, several /cmp/ review paths (such as /cmp/*/write-review) and /jobs/ browse paths with a country code or title; no crawl-delay line | The file closes the single-posting page paths; check which exact URLs you request, and keep the pace slow |
| Glassdoor (glassdoor.com) | All crawlers (*): /search/, /jobview/, /ajax/, /profile/ and several /partners/ paths disallowed, plus some /Job/ and /Reviews/ URL patterns; no crawl-delay line | Internal search and ajax paths are off limits for crawlers; read only pages the file leaves open and expect login walls on the rest |
Sources: robots.txt of indeed.com and glassdoor.com, opened 20 November 2026. The files differ per country site and change often; read the one for the site you use. This is not legal advice, and site terms may say more.
Reading one results page then following to the next is a short browsing flow: a session of 10 or 30 minutes on the Budget line keeps one IP across it, while without a session each new connection gets a new IP. Concurrent connections are not limited, but go slowly: one request every several seconds per site is a safe start.
Collect public listing data only. Do not log in to collect more, do not collect candidate or reviewer personal data, and stop on 429 or repeated checks instead of adding IPs. Store the source URL and the collection date with every posting, and remember that postings are removed or edited by employers.
import time
import urllib.robotparser
import requests
# One connection per country, copied from My assets (HTTP port 2222; the same login also works over SOCKS5 on 3333).
PROXY = "http://LOGIN:PASSWORD@proxy.proxy302.com:2222"
PROXIES = {"http": PROXY, "https": PROXY}
UA = "my-jobs-research/1.0 (contact: you@example.com)"
CHECKS = [
("https://www.indeed.com", "/jobs?q=data+analyst&l=Austin%2C+TX"),
("https://www.glassdoor.com", "/Job/austin-data-analyst-jobs-SRCH_IL.0,6_IC1139761_KO7,19.htm"),
] # replace with the public pages you are allowed to read
print("exit country:", requests.get("https://ipinfo.io/json", proxies=PROXIES, timeout=(5, 20)).json().get("country"))
for site, path in CHECKS:
rp = urllib.robotparser.RobotFileParser(site + "/robots.txt")
rp.read()
if not rp.can_fetch(UA, site + path):
print("robots.txt says no:", site + path)
continue
r = requests.get(site + path, proxies=PROXIES, headers={"User-Agent": UA}, timeout=(5, 20))
print(site + path, r.status_code, len(r.text))
time.sleep(5) # keep the pace slow; stop on 429 or a check page
Illustrative example: replace LOGIN, PASSWORD, the contact address and the two paths. For SOCKS5 install requests[socks] and use socks5h:// with port 3333.
Open Rotating Residential Proxies. Use Budget (US$1.50 per GB, country level) for country-wide job data, or Premium (US$5 per GB) when you need a state or city.
Set the Region to the country (and city on Premium) and generate a connection. Copy host, port, username and password from My assets.
Run the script above or open ipinfo.io through the proxy. Expected: the country you set.
Fetch only URLs that robots.txt leaves open, pause between requests, and save the posting URL, the market and the date.
Check Requests are too fast or too uniform, or the URL is one the site protects.
Fix Slow down, reduce parallel requests and check the URL against robots.txt; do not rotate IPs to push through.
Check The exit country or city differs from your browser location.
Fix Check the exit on ipinfo.io and set the same country (or city on Premium).
Check requests[socks] is not installed.
Fix Run python -m pip install "requests[socks]" or use the HTTP line on port 2222.
Rotating residential in the country of the Indeed site you read, from US$1.50 per GB. Check robots.txt for your exact URLs first, because single-posting paths are closed there.
Its robots.txt closes /search/, /jobview/ and ajax paths for crawlers, and its terms may say more. Read both before collecting anything.
Only when you compare local markets. Premium lets you pick a state or city; country-wide data works on Budget.
It depends on page size and volume; the GB guide shows how to estimate it.
Setup guides Proxy302 publishes on GitHub on the same topic:
Official documentation · Confirmed on:
Opened on 20 November 2026: All crawlers (*): the group opens with Allow: / and then disallows /viewjob, /rc/, /company/*, several /cmp/ review paths (such as /cmp/*/write-review) and /jobs/ browse paths with a country code or title; no crawl-delay line
robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.
Official documentation · Confirmed on:
Opened on 20 November 2026: All crawlers (*): /search/, /jobview/, /ajax/, /profile/ and several /partners/ paths disallowed, plus some /Job/ and /Reviews/ URL patterns; no crawl-delay line
robots.txt is a crawler-etiquette file and can change; it is not a legal opinion and the site terms may say more.
Official documentation · Confirmed on:
Opened on 17 November 2026 and re-read 20 November: proxies are passed as a dict to the proxies argument; user:password goes in the URL; SOCKS needs requests[socks]; socks5h resolves DNS on the proxy; requests have no timeout unless one is set.
Product observation · Confirmed on:
Product facts file of the site (dated 10 November 2026), read on 20 November 2026: per-GB prices of both rotating residential lines, traffic does not expire, countries and city targeting, protocols and ports, no free trial.
Create an account, top up once, then pick rotating residential, static ISP or datacenter IPs and have one running in your browser within minutes.