How to Scrape Indeed Jobs Data in 2026

Indeed answers a plain Python request with 403 Forbidden on the first request, from a clean address, with browser headers. No rate limit is involved and no crawl budget is spent. I sent twenty requests and got twenty 403s.

That is the fact every Indeed tutorial has to start from, and most of them skip it. This guide measures what each route returns, then covers the three that work. The no-code scraper, the JobPosting markup on the posting pages, and the Indeed endpoints that answer in JSON.

What Indeed Job Data Is Used For

Four uses cover most projects that end up here. Labour market analysis, where the counts and the skill lists matter more than any single posting. Salary research, which needs the pay range and the location together, and which is the reason the salary field decides whether a route is worth using. Recruitment, where the same query runs weekly and only the new postings matter. And competitive intelligence, where one company’s postings are tracked over months to see which teams are growing.

All four need the same three fields to be reliable, the title, the company and the location, and the first two of them decide how much the rest is worth. What follows is about getting those fields out of a site that does not want to hand them over.

What Indeed Returns to a Scraper

Indeed sits behind Cloudflare, and the wall goes up before any rate limit does. Twenty sequential requests to job search pages, browser headers, two seconds apart, one after another:

import httpx

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 15_7_7) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
}

with httpx.Client(headers=HEADERS, follow_redirects=True, timeout=30) as client:
    response = client.get("https://www.indeed.com/jobs?q=python+developer&l=New+York%2C+NY")
    print(response.status_code, len(response.text), "bytes")

The very first request returns the block page.

403 28015 bytes

The 28 KB is the challenge page rather than the job list. Rotating the query, the location and the page offset changed nothing, and neither did the two-second gap. A current User-Agent is necessary and nowhere near sufficient here, because the decision is made before the headers matter much. The TLS handshake a Python client opens differs from a browser’s in cipher order and extensions, the HTTP/2 settings frame differs too, and a datacenter address carries a reputation of its own. Those three are checked together, which is why a perfect header set from a cloud server still gets the challenge, and why the same code from a home connection sometimes does not. The blocking guide goes through what each layer sees.

The same URLs through a scraping API, three configurations, three search pages each:

Job posting pages behave differently from search pages. In a run over 30 posting URLs the same day, the rendered residential route returned 22 of them, and a smaller check an hour later returned one of three, so that route works and flaps. The dedicated endpoint returned all three, every time, in two to three seconds.

The practical reading is simple. Anything that goes through the public search pages needs the challenge solved for it. Posting pages are reachable with a rendered request from a residential exit, on a good day. The endpoints that speak JSON do not care either way, and this is what the rest of the article uses.

What a block looks like in code

A block arrives in two shapes, and only one of them raises.

response = client.get(url)
response.raise_for_status()          # 403 raises here

In a terminal that failure reads like this.

httpx.HTTPStatusError: Client error '403 Forbidden' for url 'https://www.indeed.com/jobs?q=...'

The other shape is worse. A rendered request through a proxy can return HTTP 200 with a two-megabyte challenge page, and raise_for_status is happy with it. Check for content rather than for status:

def looks_like_results(html: str) -> bool:
    markers = ("job_seen_beacon", "mosaic-provider-jobcards", "jobsearch-ResultsList")
    return any(marker in html for marker in markers) and "just a moment" not in html.lower()

A scraper without that check writes empty rows for hours and reports success.

Choosing a Method

Six routes, ordered by how much of the blocking problem they leave with you:

The rest of the article works down that table from the top.

Method 1, the No-Code Scraper

The fastest route to a spreadsheet, and the one that needs no code at all. Sign in, open No-Code Scrapers, and pick the Indeed scraper.

The form takes what the search page takes, plus a limit:

1. Results limit. 0 means every result the query returns.
2. Job title, keywords, company. One query per line, so several searches run as one job.
3. City, state, zip code. One location per line.
4. Indeed domain. The country site to search.
5. Run scraper. Billing is 10 credits per row returned.
6. Scraper tasks. Finished runs download as JSON, CSV or XLSX.

Each row carries the full description rather than the search-page summary, which is the difference between this and scraping the list yourself.

Method 2, Python Against the Pages

If the job has to run inside your own code, the choice is between a browser and an API. Both start with the same problem: the page has to arrive first.

A browser does not remove the wall

Selenium or Playwright renders JavaScript, which the search page needs, and neither changes the address the request comes from. From a datacenter address the browser gets the same challenge the HTTP client did, which is why “use Selenium” is only half an answer. Pair it with residential proxies, or use a rendering API that has its own.

from playwright.sync_api import sync_playwright

def fetch(url: str, proxy: str | None = None) -> str:
    with sync_playwright() as p:
        browser = p.chromium.launch(
            headless=True,
            proxy={"server": proxy} if proxy else None,
        )
        page = browser.new_page(user_agent=(
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 15_7_7) AppleWebKit/537.36 "
            "(KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"))
        page.goto(url, wait_until="domcontentloaded", timeout=60000)
        page.wait_for_timeout(3000)
        html = page.content()
        browser.close()
    return html

Run it against a posting URL rather than a search URL. Search pages stayed behind the challenge on every route except the dedicated endpoint, and the posting pages carry the fields worth having.

Rendering through the API

The same fetch without a local browser, and with the proxy pool included:

import requests

API_KEY = "YOUR-API-KEY"

def fetch_rendered(url: str) -> str | None:
    response = requests.post(
        "https://api.hasdata.com/scrape/web",
        json={
            "url": url,
            "proxyType": "residential",   # datacenter did not get past the challenge here
            "proxyCountry": "US",
            "jsRendering": True,
            "wait": 3000,
            "outputFormat": ["html"],
        },
        headers={"x-api-key": API_KEY, "Content-Type": "application/json"},
        timeout=180,
    )
    if response.status_code != 200:
        print("no page:", response.status_code)
        return None
    return response.text

A rendered request on residential exits costs 15 credits, against 1 for a plain datacenter fetch, and on this target the cheaper configurations returned nothing at all.

Reading the JobPosting Markup

Once a posting page is in hand, the fields are already structured. Google requires JobPosting markup for a posting to appear in its jobs results, so Indeed publishes it as JSON-LD in the page head, and parsing that beats writing selectors: the block does not change when the layout does.

import json
from bs4 import BeautifulSoup


def job_postings(html: str) -> list[dict]:
    """Every JobPosting object in the page, including the ones inside @graph."""
    soup = BeautifulSoup(html, "html.parser")
    found = []
    for script in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(script.string or "")
        except json.JSONDecodeError:
            continue
        items = data if isinstance(data, list) else [data]
        for item in list(items):
            if isinstance(item, dict) and "@graph" in item:
                items.extend(item["@graph"])
        found.extend(i for i in items if isinstance(i, dict) and i.get("@type") == "JobPosting")
    return found


def flatten(posting: dict) -> dict:
    org = posting.get("hiringOrganization") or {}
    place = posting.get("jobLocation") or {}
    if isinstance(place, list):
        place = place[0] if place else {}
    address = place.get("address") or {}
    salary = (posting.get("baseSalary") or {}).get("value") or {}
    return {
        "title": posting.get("title"),
        "company": org.get("name") if isinstance(org, dict) else org,
        "city": address.get("addressLocality") if isinstance(address, dict) else None,
        "region": address.get("addressRegion") if isinstance(address, dict) else None,
        "remote": posting.get("jobLocationType") == "TELECOMMUTE",
        "employment_type": posting.get("employmentType"),
        "salary_min": salary.get("minValue"),
        "salary_max": salary.get("maxValue"),
        "salary_unit": salary.get("unitText"),
        "date_posted": posting.get("datePosted"),
        "valid_through": posting.get("validThrough"),
    }


html = fetch_rendered("https://www.indeed.com/m/viewjob?jk=73f8ea23bde71d96")
if html:
    for posting in job_postings(html):
        print(flatten(posting))

How much this buys on Indeed, measured on the postings that came back in a 30-page run: 21 of the 22 readable pages carried a JobPosting block, 20 of those carried baseSalary, and a salary was visible in the page text on 14. The markup was the more complete source, and it needed no selector at all. The fields present on a returned posting were title, hiringOrganization, jobLocation, baseSalary, datePosted, validThrough, description and directApply.

validThrough deserves a mention of its own. It is the expiry Indeed reports to Google, which makes it a better freshness signal than the “posted 3 days ago” string on the card, and it is what a daily collector should filter on. The general technique, and the same measurement across ten job boards, is in the guide to scraping job postings.

Method 3, the Indeed APIs

Two endpoints cover the two shapes of the job, a search and a posting. Both answer in JSON, both cost 5 credits per call, and neither has to get past the challenge, because that is the service’s problem rather than yours.

Listings

One call takes the same keyword and location the search box takes, and returns the page of results as objects:

import requests

API_KEY = "YOUR-API-KEY"

response = requests.get(
    "https://api.hasdata.com/scrape/indeed/listing",
    params={"keyword": "python developer", "location": "New York, NY", "domain": "www.indeed.com"},
    headers={"x-api-key": API_KEY},
    timeout=120,
)
jobs = response.json().get("jobs", [])
print(len(jobs), "jobs")
for job in jobs[:3]:
    print(f"{job['title']} | {job['company']} | {job['location']} | {job.get('salary') or 'no salary'}")

One call for that query returned 53 jobs in six seconds, with a salary on 48 of them and 28 marked as sponsored:

53 jobs
Python Developer | Think IT Technologies | New York, NY 10114 | $60 - $70 an hour
Python Developer | Infosys | New York, NY | $95,000 - $130,000 a year
Senior Python Engineer | Capital One | New York, NY | no salary

Each row carries title, company, location, date, isoDate, salary, sponsored, description, details and url. The url is what the second endpoint takes.

Depth comes from start, which moves the window the same way the search page does. Twelve calls on one query returned 328 jobs with no repeats at all, 293 of them with a salary:

import time

def all_jobs(keyword: str, location: str, max_calls: int = 12) -> list[dict]:
    seen, start = {}, 0
    for _ in range(max_calls):
        response = requests.get(
            "https://api.hasdata.com/scrape/indeed/listing",
            params={"keyword": keyword, "location": location, "start": start},
            headers={"x-api-key": API_KEY},
            timeout=120,
        )
        if response.status_code != 200:
            break
        jobs = response.json().get("jobs", [])
        new = [j for j in jobs if j.get("url") not in seen]
        seen.update({j["url"]: j for j in new})
        print(f"start={start}: {len(jobs)} jobs, {len(new)} new, {len(seen)} unique")
        if not new:
            break
        start += 10
        time.sleep(1)
    return list(seen.values())

The counter confirms that every early page still adds only new jobs.

start=0: 53 jobs, 53 new, 53 unique
start=10: 25 jobs, 25 new, 78 unique
start=20: 25 jobs, 25 new, 103 unique
...
start=110: 25 jobs, 25 new, 328 unique

That run cost 60 credits for 328 jobs, and the loop stops on its own when a page stops adding anything, which is the condition to keep rather than a fixed page count.

One posting in full

The second endpoint takes the URL from a listing row and returns the whole posting, description included:

import requests

API_KEY = "YOUR-API-KEY"

response = requests.get(
    "https://api.hasdata.com/scrape/indeed/job",
    params={"url": "https://www.indeed.com/m/viewjob?jk=73f8ea23bde71d96"},
    headers={"x-api-key": API_KEY},
    timeout=120,
)
job = response.json()
print(job["title"], "|", job["company"], "|", job.get("salary"))
print(job["description"][:200])

Three calls to three postings returned all three, in 1.9 to 3.4 seconds, with title, company, location, salary, details, sponsored, description and descriptionHtml. That is the same field set the markup carries, without the rendering step and without the flapping.

Both endpoints can be tried from the playground with parameters filled in, which is faster than writing the first call by hand.

Saving the Data

json and csv are part of the standard library, so nothing to install. One function writes both, and the CSV keeps the columns you actually query on:

import csv
import json


def save(jobs: list[dict], stem: str = "indeed_jobs") -> None:
    with open(f"{stem}.json", "w", encoding="utf-8") as f:
        json.dump(jobs, f, ensure_ascii=False, indent=2)

    if not jobs:
        return
    columns = ["title", "company", "location", "salary", "date", "url"]
    with open(f"{stem}.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=columns, extrasaction="ignore")
        writer.writeheader()
        writer.writerows(jobs)
    print(f"saved {len(jobs)} jobs")

extrasaction="ignore" is the part worth copying. Job records carry a long description and a nested details object, and without it the writer raises on the first row that has a field the header does not list.

For a collector that runs daily, keep the IDs you have already seen and filter on validThrough before writing, which the job scraping guide covers with a working state file.

Conclusion

Indeed is not a scraping tutorial target. A plain request gets 403 on the first try, a browser from the wrong address gets the same challenge, and the search pages stayed behind that challenge on every route measured here except the dedicated endpoint.

Three things work, in the order most projects should try them. The Indeed APIs when you want JSON and no maintenance, the no-code scraper when the result is a spreadsheet for someone else, and a rendered request through residential exits plus the JobPosting markup when you need the raw page for something the endpoints do not return.