How to Scrape Job Postings with Python in 2026

Job postings are the most structured content on the web. Google only lists a posting in its jobs results when the page carries JobPosting markup, so Indeed, Glassdoor, Dice and most boards publish the title, company, location, salary, posting date and expiry as JSON-LD in the page head, though not every board does it on every page, as the measurement below shows. A job scraper that reads that block first gets clean fields without a single CSS selector and falls back to selectors only where the markup is missing. I fetched 300 postings from ten boards for this guide, and 181 of them carried the markup, 87 of those with a salary range, against 91 pages that showed a salary in the visible text.

This guide builds a ZipRecruiter scraper in Python that does both, then covers pagination, a schedule that collects new postings every day, and what boards return when they decide a client is a bot. Rendering and proxies go through the Web Scraping API, which returns the page HTML. The parsing stays in your code.

Identify Target Data on Job Boards

Before writing code, you need to understand what data you want to extract and where it’s located on the page.

Open ZipRecruiter (or any other board) in your browser and search for a job (for example, “marketing manager” in “New York, NY”). You’ll see a list of job cards, each containing a title, company name, location, and salary.

Right-click on any job title and select “Inspect” (or press F12). This opens Developer Tools showing the HTML structure. You’ll see that each job is wrapped in an `<article>` tag. The title is in an `<h2>` tag, and company name has a special attribute `data-testid="job-card-company"`.

These patterns are called CSS selectors. They tell your code exactly where to find data. For ZipRecruiter, the main selectors are:

Each selector points to a specific piece of information. The posting page behind each card also carries the same fields in its head, in a form that does not change with the layout, and the section after the setup reads that first.

Environment Setup and Dependencies

Install Python (version 3.13 or higher) and the required libraries. Open your terminal and run:

pip install requests beautifulsoup4

These two libraries are all you need:

• `requests` sends HTTP requests to the HasData API
• `beautifulsoup4` parses the HTML and extracts data

Why HasData API?

ZipRecruiter uses JavaScript to load job listings dynamically. Simple HTTP requests won’t work because you’ll get an empty page, the content loads after the page opens. You need a tool that renders JavaScript like a real browser.

The Web Scraping API renders the page in a browser on its side, waits for JavaScript to finish, and returns the HTML. Each request goes out through a datacenter or residential proxy in the country you ask for, and the response is the page as the browser saw it, including whatever the board decided to show that client.

Get Your API Key

Sign up at hasdata.com and copy the API key from the API Keys page of the dashboard.

Store your API key safely. You’ll use it in every request:

HASDATA_API_KEY = "HASDATA-API-KEY"
HASDATA_API_URL = "https://api.hasdata.com/scrape/web"

That’s it for setup. The scraper starts on the posting page, where the cleanest fields are the ones the board publishes for Google.

Read the JobPosting Markup Before Writing Selectors

Open a posting page and search the source for application/ld+json. The block with "@type": "JobPosting" is the schema.org object boards publish for Google Jobs, and it holds title, hiringOrganization, jobLocation, baseSalary, datePosted, validThrough, employmentType and the full description as HTML. It is the same trick as parsing Walmart’s Product markup, applied to a page type where the markup is close to mandatory. Search-result pages rarely carry one object per card, so the listing scraper below still needs selectors for the cards, and the posting pages behind the cards are where the markup pays off.

The function below fetches a posting through the Web Scraping API with a datacenter proxy and no rendering (1 credit), parses every JSON-LD block on the page, including @graph arrays, and flattens the first JobPosting into the fields a scraper wants:

import json

import requests
from bs4 import BeautifulSoup

HASDATA_API_KEY = "HASDATA-API-KEY"
HASDATA_API_URL = "https://api.hasdata.com/scrape/web"


def fetch_page(url):
    """Fetch a page through the Web Scraping API (datacenter proxy, no rendering: 1 credit)."""
    payload = {"url": url, "proxyType": "datacenter", "proxyCountry": "US", "outputFormat": ["html"]}
    headers = {"x-api-key": HASDATA_API_KEY, "Content-Type": "application/json"}
    response = requests.post(HASDATA_API_URL, json=payload, headers=headers, timeout=120)
    response.raise_for_status()
    return response.text  # with outputFormat ["html"] the API returns the page itself, not a JSON wrapper


def job_postings(html):
    """Return every JobPosting object found in the page's JSON-LD blocks."""
    soup = BeautifulSoup(html, "html.parser")
    found = []
    for script in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(script.string or "")
        except json.JSONDecodeError:
            continue
        items = data if isinstance(data, list) else [data]
        for item in list(items):
            if isinstance(item, dict) and "@graph" in item:
                items.extend(item["@graph"])
        found.extend(i for i in items if isinstance(i, dict) and i.get("@type") == "JobPosting")
    return found


def flatten(posting):
    """Pick the fields a job scraper usually wants out of a JobPosting."""
    org = posting.get("hiringOrganization") or {}
    place = posting.get("jobLocation") or {}
    if isinstance(place, list):
        place = place[0] if place else {}
    address = place.get("address") or {}
    salary = posting.get("baseSalary") or {}
    value = salary.get("value") or {}
    return {
        "title": posting.get("title"),
        "company": org.get("name") if isinstance(org, dict) else org,
        "city": address.get("addressLocality") if isinstance(address, dict) else address,
        "region": address.get("addressRegion") if isinstance(address, dict) else None,
        "remote": posting.get("jobLocationType") == "TELECOMMUTE",
        "employment_type": posting.get("employmentType"),
        "salary_min": value.get("minValue") if isinstance(value, dict) else None,
        "salary_max": value.get("maxValue") if isinstance(value, dict) else None,
        "salary_unit": value.get("unitText") if isinstance(value, dict) else None,
        "date_posted": posting.get("datePosted"),
        "valid_through": posting.get("validThrough"),
    }


if __name__ == "__main__":
    url = "https://www.dice.com/job-detail/6f57c501-b868-4870-861c-49a6420b1020"
    postings = job_postings(fetch_page(url))
    print(f"{len(postings)} JobPosting object(s) on the page")
    for posting in postings:
        print(json.dumps(flatten(posting), indent=2, ensure_ascii=False))

On a Dice posting from the run for this guide:

1 JobPosting object(s) on the page
{
  "title": "Python Developer",
  "company": "Codeforce 360",
  "city": null,
  "region": null,
  "remote": true,
  "employment_type": "CONTRACTOR",
  "salary_min": null,
  "salary_max": null,
  "salary_unit": null,
  "date_posted": "2026-08-26T13:43:09.000Z",
  "valid_through": "2026-09-26T13:43:09.000Z"
}

This posting carries neither a location nor a salary, which is what the null fields mean, and 27 of the 30 Dice postings in the run did carry baseSalary. Salary in the markup is a MonetaryAmount with minValue, maxValue and unitText (HOUR, YEAR), so the parse_salary regex further down is only needed for boards that leave baseSalary out. validThrough is the expiry the board reports to Google, which is a better freshness signal than a “New” badge.

How Often the Markup Is There

I took 30 posting URLs from each of ten sources, 300 in all, for five occupations from Python developer to registered nurse. Indeed and Glassdoor came from their listing endpoints, LinkedIn, ZipRecruiter, Dice, Built In and Monster from Google results, and three applicant-tracking systems that host company career pages (Greenhouse, Lever, Workday) from Google results as well. Each URL was fetched through the Web Scraping API with a datacenter proxy and no rendering first, the 1-credit route, and again with rendering and a residential proxy when the first fetch failed or came back as a challenge or login page. The parser looked for a JobPosting object in any JSON-LD block, checked whether it carried baseSalary, and compared its title with the page’s H1. Two selector-style checks ran on the same HTML as well, an H1 present and a salary pattern in the visible text.

The same counts drawn per board put the three salary signals next to each other.

The plain fetch returned the page on 147 of 300 postings, all of them on LinkedIn, Dice, Built In, Lever and Greenhouse. The other 153 went through the rendered residential route after a 200 that was a shell or a challenge page (no board answered with a 403), and 103 of those came back with a page or with the markup. With both routes, 248 of the 300 URLs were readable, and 181 of them (73%) carried a JobPosting object. Every readable page on Dice, Built In and Indeed had one, Lever and Workday 25 of 30 each. Greenhouse had it on 2 of 30 (the ATS leaves the markup to the company’s own site), LinkedIn on 8 of 30 (the guest view of a posting carries it only sometimes), ZipRecruiter on 15 of 30. Monster returned an empty body on both routes for all 30 URLs, and 8 Indeed URLs answered 400 on both routes, so those are outside the count.

The markup is richer than the page on salary. 87 of the 181 objects carried baseSalary, while a salary pattern appeared in the visible text of 91 of 248 readable pages, and on Dice, Indeed and Glassdoor the markup had a salary more often than the page showed one. Where a page had both an H1 and the markup, the two titles agreed on 128 of 131, and the 3 disagreements were Glassdoor error pages whose H1 read “This page couldn’t load” while the JobPosting block was still in the head. 50 postings had the markup and no H1 at all, every Lever and Workday page among them, so the selector path had nothing to anchor on where the markup was complete. Field coverage inside the 181 objects: title on 181, hiringOrganization on 181, datePosted on 181, description on 181, jobLocation on 172, employmentType on 168, validThrough on 131, identifier on 110.

Implementing the Extraction Logic

The process has three stages. Fetch the page, parse the HTML, then extract the fields from each job card.

Fetching Pages with HasData API

First, create a function that fetches rendered HTML through HasData API:

def fetch_page(url):
    """Fetch page through HasData API"""
    headers = {
        "x-api-key": HASDATA_API_KEY,
        "Content-Type": "application/json"
    }
    
    payload = {
        "url": url,
        "proxyType": "residential",
        "proxyCountry": "US",
        "jsRendering": True,
        "blockAds": True,
        "outputFormat": ["html"]
    }
    
    response = requests.post(HASDATA_API_URL, json=payload, headers=headers)
    return response.text

This function sends a POST request to HasData API with the target URL and configuration. The API returns fully rendered HTML with all JavaScript content loaded. Setting `jsRendering: True` ensures dynamic content appears, and `proxyType: "residential"` routes the request through a residential exit, which costs 15 credits per rendered page against 10 on datacenter proxies.

Parsing Job Cards

Once you have the HTML, use BeautifulSoup to find all job listings:

soup = BeautifulSoup(html, 'html.parser')
job_cards = soup.find_all('article')
print(f"Found {len(job_cards)} jobs")

ZipRecruiter wraps each job posting in an `<article>` tag. This selector finds all job cards on the page at once. If you see `0 jobs`, the page didn’t load correctly. If you see a number like `20-40`, you’re successfully finding the cards.

Extracting Data Fields

For each job card, extract all available information:

def extract_job_data(card):
    """Extract all data from a single job card"""
    # Job ID
    job_id = card.get('id', '').replace('job-card-', '')
    
    # Title
    title_elem = card.find('h2')
    title = title_elem.get_text(strip=True) if title_elem else 'N/A'
    
    # Company
    company_elem = card.find('a', {'data-testid': 'job-card-company'})
    company = company_elem.get_text(strip=True) if company_elem else 'N/A'
    
    # Location
    location_elem = card.find('a', {'data-testid': 'job-card-location'})
    location = location_elem.get_text(strip=True) if location_elem else 'N/A'
    
    # Salary - search paragraphs for dollar signs
    salary = 'N/A'
    for p in card.find_all('p'):
        text = p.get_text(strip=True)
        if '$' in text and '/' in text:
            salary = text
            break
    
    # Company logo
    logo_elem = card.find('img')
    logo_url = logo_elem.get('src') if logo_elem else None
    
    # Badges
    badges = []
    if card.find('p', string='New'):
        badges.append('New')
    if card.find('p', string='Quick apply'):
        badges.append('Quick apply')

Use `data-testid` attributes when available because they’re more stable than CSS classes. Always check if an element exists before calling `.get_text()` to avoid errors when data is missing. The `strip=True` parameter removes extra whitespace. Return `'N/A'` for missing data to keep your data structure consistent.

For salary, search through all paragraph tags looking for text containing both `$` and `/` to match formats like `$80K/yr` or `$25/hr`. The loop breaks after finding the first match to avoid picking up other dollar amounts.

Data Cleaning and Standardization

Raw scraped data needs cleaning before it’s useful. Convert URLs, separate location components, and structure the output.

Converting Relative to Absolute URLs

Company links on ZipRecruiter are relative URLs that start with `/`. Convert them to complete URLs:

company_url = company_elem.get('href') if company_elem else None
if company_url and not company_url.startswith('http'):
    company_url = 'https://www.ziprecruiter.com' + company_url

This transforms `/co/Acme/Jobs` into `https://www.ziprecruiter.com/co/Acme/Jobs`. Check if the URL already starts with `http` to avoid breaking external links.

Separating Location and Remote Status

Location data includes both city/state and remote status in one field. Parse them separately:

location_elem = card.find('a', {'data-testid': 'job-card-location'})
location = 'N/A'
is_remote = False


if location_elem:
    # Get parent element to check for "· Remote" text
    location_parent = location_elem.parent
    full_text = location_parent.get_text(strip=True)
    
    # Check if remote
    is_remote = 'Remote' in full_text
    
    # Get just the city/state
    location = location_elem.get_text(strip=True)

The location link contains only the city and state like `Austin, TX`. The remote indicator appears as separate text `· Remote` next to it. Checking the parent element’s full text catches this. Now you have both `location: "Austin, TX"` and `is_remote: true` as separate fields, making it easy to filter for remote-only jobs later.

Building the Complete Job Dictionary

Combine all extracted and cleaned data into a structured dictionary:

return {
    'job_id': job_id,
    'title': title,
    'company': company,
    'company_url': company_url,
    'location': location,
    'is_remote': is_remote,
    'salary': salary,
    'logo_url': logo_url,
    'badges': badges
}

This structure makes the data easy to filter and analyze. You can search for remote jobs with `is_remote == True`, filter by company, or sort by salary.

Handling Pagination and Scale

Most searches return multiple pages of results. Loop through pages automatically until you reach the end.

The page number sits in the URL path, so the loop builds each URL directly instead of clicking the button.

Building Pagination URLs

ZipRecruiter uses a specific URL pattern for pages:

query_formatted = query.replace(' ', '+')
location_formatted = location.replace(' ', '+')


if page == 1:
    url = f"https://www.ziprecruiter.com/jobs-search?search={query_formatted}&location={location_formatted}"
else:
    url = f"https://www.ziprecruiter.com/jobs-search/{page}?search={query_formatted}&location={location_formatted}"

Page 1 uses `/jobs-search?search=...` while pages 2+ use `/jobs-search/2?search=...`. Replace spaces with `+` signs in the query and location parameters.

Example URLs:

• Page 1: `/jobs-search?search=python+developer&location=Remote`
• Page 2: `/jobs-search/2?search=python+developer&location=Remote`
• Page 3: `/jobs-search/3?search=python+developer&location=Remote`

Detecting the Last Page

Check for the next page button to know when to stop:

for page in range(1, max_pages + 1):
    # Fetch and parse page...
    
    # Check if more pages exist
    next_button = soup.find('a', {'title': 'Next Page'})
    if not next_button:
        print("No more pages available")
        break
    
    # Wait before next request
    time.sleep(2)

The next page button disappears on the last page. When `find()` returns `None`, stop the loop. Add a 2-second delay with `time.sleep(2)` between requests to be respectful to the server and avoid rate limiting.

Collecting Continuously

A one-off run answers “what is open today”. A feed of new postings, the thing a job tracker or a market analysis needs, is the same scraper run on a schedule with a memory of what it already saw. Keep the job IDs from earlier runs in a file, keep only the postings whose ID is new, and drop the ones whose validThrough from the markup has passed. Run it from cron or a GitHub Actions schedule once a day. Boards repost the same job under a new ID after a few weeks, so a second key of company plus normalized title catches most reposts.

import json
import re
from datetime import date
from pathlib import Path

STATE_FILE = Path("seen_jobs.json")


def repost_key(job):
    """Company plus normalized title: catches the same opening reposted under a new ID."""
    text = f"{job.get('company', '')} {job.get('title', '')}".lower()
    return re.sub(r"[^a-z0-9]+", " ", text).strip()


def new_postings(jobs, today=None):
    """Keep only postings not seen in earlier runs and not past their validThrough date, then remember them."""
    today = today or date.today().isoformat()
    state = json.loads(STATE_FILE.read_text()) if STATE_FILE.exists() else {"ids": [], "keys": []}
    seen_ids, seen_keys = set(state["ids"]), set(state["keys"])
    fresh = []
    for job in jobs:
        expired = job.get("valid_through") and job["valid_through"][:10] < today
        if expired or job.get("job_id") in seen_ids or repost_key(job) in seen_keys:
            continue
        fresh.append(job)
        seen_ids.add(job.get("job_id"))
        seen_keys.add(repost_key(job))
    STATE_FILE.write_text(json.dumps({"ids": sorted(i for i in seen_ids if i), "keys": sorted(seen_keys)}, indent=2))
    return fresh


if __name__ == "__main__":
    # after scrape_jobs(...) from the complete scraper:
    jobs = json.loads(Path("jobs_full.json").read_text(encoding="utf-8"))
    fresh = new_postings(jobs)
    print(f"{len(jobs)} postings scraped, {len(fresh)} new since the last run")
    Path(f"new_jobs_{date.today().isoformat()}.json").write_text(json.dumps(fresh, indent=2, ensure_ascii=False), encoding="utf-8")

Run twice on the same three test postings, the function reported one new posting the first time (one of the three had a validThrough in the past and one was the same job under a second ID) and none the second time. Deduplicating across boards is a separate step, because the same opening appears on Indeed, LinkedIn and the company’s Greenhouse page with three different IDs. The company name and the title from the JobPosting markup, lowercased and stripped of punctuation, make a workable cross-board key. The first page of each search is enough for a daily run once the memory file exists, because new postings sort to the top on every major board.

Saving Results

After scraping, save the data to a file for later use or analysis.

Exporting to JSON

JSON format preserves the nested structure of your data:

with open('jobs_full.json', 'w', encoding='utf-8') as f:
    json.dump(jobs, f, indent=2, ensure_ascii=False)


print(f"Saved {len(jobs)} jobs to jobs_full.json")

The `indent=2` parameter makes the file readable. Set `ensure_ascii=False` to properly handle company names with special characters. The resulting file looks like:

[
  {
    "job_id": "hwgl32RTP2J7yXQ1o5EmTw",
    "title": "Software Engineer, Full Stack",
    "company": "ZipRecruiter",
    "company_url": "https://www.ziprecruiter.com/co/ZipRecruiter/Jobs/--in-Remote?uuid=qWAzEkSdWiDDN9H1IU6DH0KAE20%3D&radius=25",
    "location": "Palo Alto, CA",
    "is_remote": true,
    "salary": "$105K - $145K/yr",
    "logo_url": "https://www.ziprecruiter.com/svc/fotomat/public-nosensitive-ziprecruiter-logos/company/314d8bb3.png",
    "is_new": false,
    "quick_apply": true,
    "be_seen_first": false,
    "badges": [
      "Quick apply"
    ]
  },
…]

Each object is one posting and badges stays a list, which the CSV export flattens into a string.

Exporting to CSV

CSV format works well with Excel and data analysis tools:

import csv


with open('jobs_full.csv', 'w', newline='', encoding='utf-8') as f:
    if jobs:
        writer = csv.DictWriter(f, fieldnames=jobs[0].keys())
        writer.writeheader()
        writer.writerows(jobs)


print(f"Saved {len(jobs)} jobs to jobs_full.csv")

CSV files flatten nested data. The `badges` list becomes a string like `['New', 'Quick apply']`. Use `newline=''` to avoid extra blank rows on Windows.

Complete Working Job Scraper

Here’s the full scraper that combines all the pieces:

import requests
from bs4 import BeautifulSoup
import json
import time


# Configuration
HASDATA_API_KEY = "HASDATA-API-KEY"
HASDATA_API_URL = "https://api.hasdata.com/scrape/web"


def fetch_page(url):
    """Fetch page through HasData API"""
    headers = {
        "x-api-key": HASDATA_API_KEY,
        "Content-Type": "application/json"
    }
    
    payload = {
        "url": url,
        "proxyType": "residential",
        "proxyCountry": "US",
        "jsRendering": True,
        "blockAds": True,
        "outputFormat": ["html"]
    }
    
    response = requests.post(HASDATA_API_URL, json=payload, headers=headers)
    return response.text


def extract_job_data(card):
    """Extract all data from a single job card"""
    # Job ID from article tag
    job_id = card.get('id', '').replace('job-card-', '') if card.get('id') else None
    
    # Title
    title_elem = card.find('h2')
    title = title_elem.get_text(strip=True) if title_elem else 'N/A'
    
    # Company
    company_elem = card.find('a', {'data-testid': 'job-card-company'})
    company = company_elem.get_text(strip=True) if company_elem else 'N/A'
    company_url = company_elem.get('href') if company_elem else None
    if company_url and not company_url.startswith('http'):
        company_url = 'https://www.ziprecruiter.com' + company_url
    
    # Location and Remote status
    location_elem = card.find('a', {'data-testid': 'job-card-location'})
    location = 'N/A'
    is_remote = False
    
    if location_elem:
        # Get the full location text including remote
        location_parent = location_elem.parent
        full_location_text = location_parent.get_text(strip=True) if location_parent else location_elem.get_text(strip=True)
        
        # Check if remote
        is_remote = 'Remote' in full_location_text
        
        # Get just the location without "· Remote"
        location = location_elem.get_text(strip=True)
    
    # Salary
    salary = 'N/A'
    for p in card.find_all('p'):
        text = p.get_text(strip=True)
        if '$' in text and '/' in text:  # Make sure it's salary format
            salary = text
            break
    
    # Company logo
    logo_elem = card.find('img')
    logo_url = logo_elem.get('src') if logo_elem else None
    
    # Badges
    badges = []
    is_new = False
    quick_apply = False
    be_seen_first = False
    
    # Check for "New" badge
    new_badge = card.find('p', string='New')
    if new_badge:
        is_new = True
        badges.append('New')
    
    # Check for "Quick apply" badge
    quick_apply_elem = card.find('p', string='Quick apply')
    if quick_apply_elem:
        quick_apply = True
        badges.append('Quick apply')
    
    # Check for "Be Seen First" badge
    be_seen_elem = card.find('p', string='Be Seen First')
    if be_seen_elem:
        be_seen_first = True
        badges.append('Be Seen First')
    
    return {
        'job_id': job_id,
        'title': title,
        'company': company,
        'company_url': company_url,
        'location': location,
        'is_remote': is_remote,
        'salary': salary,
        'logo_url': logo_url,
        'is_new': is_new,
        'quick_apply': quick_apply,
        'be_seen_first': be_seen_first,
        'badges': badges
    }


def scrape_jobs(query, location, max_pages=3):
    """Scrape multiple pages of job listings"""
    all_jobs = []
    
    # Format query and location for URL
    query_formatted = query.replace(' ', '+')
    location_formatted = location.replace(' ', '+')
    
    for page in range(1, max_pages + 1):
        print(f"\n--- Scraping page {page} ---")
        
        # Build URL
        if page == 1:
            url = f"https://www.ziprecruiter.com/jobs-search?search={query_formatted}&location={location_formatted}"
        else:
            url = f"https://www.ziprecruiter.com/jobs-search/{page}?search={query_formatted}&location={location_formatted}"
        
        print(f"URL: {url}")
        
        # Fetch page
        html = fetch_page(url)
        soup = BeautifulSoup(html, 'html.parser')
        
        # Find job cards
        job_cards = soup.find_all('article')
        print(f"Found {len(job_cards)} jobs")
        
        if len(job_cards) == 0:
            print("No jobs found, stopping")
            break
        
        # Extract data from each card
        for card in job_cards:
            job = extract_job_data(card)
            all_jobs.append(job)
        
        # Check for next page
        next_button = soup.find('a', {'title': 'Next Page'})
        if not next_button:
            print("No next page button found")
            break
        
        # Wait before next page
        if page < max_pages:
            print("Waiting 2 seconds...")
            time.sleep(2)
    
    return all_jobs


def main():
    """Main function"""
    print("=" * 60)
    print("ZipRecruiter Job Scraper")
    print("=" * 60)
    
    # Scrape jobs
    jobs = scrape_jobs(
        query="python developer",
        location="Remote",
        max_pages=3
    )
    
    # Save to JSON
    with open('jobs_full.json', 'w', encoding='utf-8') as f:
        json.dump(jobs, f, indent=2, ensure_ascii=False)
    
    print(f"\nSaved {len(jobs)} jobs to jobs_full.json")
    
    # Show summary
    print(f"\n--- Summary ---")
    print(f"Total jobs: {len(jobs)}")
    print(f"Remote jobs: {sum(1 for j in jobs if j['is_remote'])}")
    print(f"Jobs with salary: {sum(1 for j in jobs if j['salary'] != 'N/A')}")
    print(f"New jobs: {sum(1 for j in jobs if j['is_new'])}")
    print(f"Quick apply: {sum(1 for j in jobs if j['quick_apply'])}")
    print(f"Be Seen First: {sum(1 for j in jobs if j['be_seen_first'])}")
    
    # Show first job
    if jobs:
        print(f"\n--- First Job ---")
        print(json.dumps(jobs[0], indent=2))


if __name__ == "__main__":
    main()

Run the scraper to get output like:

============================================================
ZipRecruiter Job Scraper
============================================================

--- Scraping page 1 ---
URL: https://www.ziprecruiter.com/jobs-search?search=python+developer&location=Remote
Found 41 jobs
Waiting 2 seconds...

--- Scraping page 2 ---
URL: https://www.ziprecruiter.com/jobs-search/2?search=python+developer&location=Remote
Found 41 jobs
Waiting 2 seconds...

--- Scraping page 3 ---
URL: https://www.ziprecruiter.com/jobs-search/3?search=python+developer&location=Remote
Found 41 jobs

Saved 123 jobs to jobs_full.json

--- Summary ---
Total jobs: 123
Remote jobs: 86
Jobs with salary: 68
New jobs: 12
Quick apply: 102
Be Seen First: 6

--- First Job ---
{
  "job_id": "sy791XyFrG2ojPTK-2Cc0Q",
  "title": "Sr. Lead Machine Learning Engineer",
  "company": "Capital One",
  "company_url": "https://www.ziprecruiter.com/co/Capital-One/Jobs/-in-Manhattan,NY?uuid=LvOba8F2oEpGlw2IxG%2FRnmjRZMM%3D&radius=25",
  "location": "Manhattan, NY",
  "is_remote": false,
  "salary": "N/A",
  "logo_url": "https://www.ziprecruiter.com/svc/fotomat/public-nosensitive-ziprecruiter-logos/company/60122984.png",
  "is_new": false,
  "quick_apply": false,
  "be_seen_first": false,
  "badges": []
}

The scraper collects jobs from multiple pages, handles pagination automatically, and saves everything to a JSON file ready for analysis. The first job of the run shows the selector path’s weak spot: a posting with no salary on the card, where the JobPosting block on the posting page would still carry one if the employer filed it.

Common Challenges When Scraping Job Boards

Job sites present unique challenges that go beyond typical web scraping. Here are the main issues you’ll encounter and how to handle them.

JavaScript-Heavy Content

Most modern job boards load listings dynamically with JavaScript. When you fetch a page with basic HTTP requests, you get an empty skeleton without any job data.

The problem. ZipRecruiter, Indeed, and similar sites render content after the page loads. A simple `requests.get()` returns HTML with no job cards visible.

The solution. Render the page. The Web Scraping API does it with jsRendering: True, or you drive a browser yourself with Playwright or Selenium. The Playwright version of fetch_page waits for the first job card and hands the rendered HTML to the same BeautifulSoup code:

from playwright.sync_api import sync_playwright


def fetch_page_playwright(url, card_selector="article"):
    """Render the page in a local Chromium and return the HTML once the first job card is in the DOM."""
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="domcontentloaded", timeout=60000)
        page.wait_for_selector(card_selector, timeout=30000)
        html = page.content()
        browser.close()
    return html

From a home connection, the same function waited the full 30 seconds for an article on the ZipRecruiter search page and timed out, because the page the browser received was a challenge page rather than the listing. On books.toscrape.com it returned 20 cards in a few seconds. A local browser is the route for boards that let a browser in, and for the rest the rendered fetch through the API is what produced the pages in this guide.

Frequent Layout Changes

Job sites update their HTML structure regularly. A selector that works today might break next week when they redesign a section.

The problem. CSS classes like `.job-card-v2-wrapper` change to `.job-listing-container` after an update. Your scraper stops finding jobs.

The solution. Read the JobPosting markup for the fields it carries, because a board keeps it stable to stay in Google Jobs while the visible layout changes around it. For the rest, use `data-testid` attributes when available, sites change these less often. If a selector breaks, check the current HTML structure in DevTools and update your code. Keep selectors simple and specific. Instead of targeting five nested div classes, use unique attributes like `data-testid='job-card-company'`.

Missing Data Fields

Not all job postings include every field. Some lack salary information, others don’t specify if they’re remote.

The problem. Calling `.get_text()` on a `None` element crashes your scraper.

The solution. Always check if an element exists before extracting text:

salary_elem = card.find('p', string=lambda x: x and '$' in x)
salary = salary_elem.get_text(strip=True) if salary_elem else 'N/A'

Return `'N/A'` for missing fields instead of `None` or skipping the job entirely. This keeps your data structure consistent and makes it easier to filter results later.

Duplicate Job Listings

Job boards often show the same posting multiple times, once from the company directly and again from recruiting agencies.

The problem. Your dataset contains duplicate jobs with slightly different titles or company names.

The solution. Track job IDs to filter duplicates:

seen_ids = set()
unique_jobs = []


for job in all_jobs:
    if job['job_id'] not in seen_ids:
        seen_ids.add(job['job_id'])
        unique_jobs.append(job)


print(f"Removed {len(all_jobs) - len(unique_jobs)} duplicates")

Use the `job_id` from the article tag’s `id` attribute, it’s unique per posting. If no ID exists, create one from the title and company name combined.

Blocks, CAPTCHAs and Login Walls

Job boards answer a plain HTTP client in three ways other than with the page. Some return 403 outright, some return a 200 with a challenge page (Cloudflare’s “Just a moment”, an Akamai or PerimeterX interstitial), and some return a login wall with the job hidden behind it. All three parse as HTML, so a scraper that only checks the status code writes empty rows.

On the 300 postings measured for this guide, 147 answered a plain request with the page and none answered with a 403. The other 153 answered 200 with something other than the posting. Indeed, Glassdoor and ZipRecruiter sent a challenge or login page, Workday an application shell with the data still to load, Monster an empty body. The rendered residential route turned 103 of those 153 into a readable page or at least the markup. Monster’s 30 stayed empty on both routes and 8 Indeed URLs answered 400 either way.

Detect the three cases before parsing. A status other than 200, a body shorter than a few hundred characters, or a <title> that says “Just a moment”, “Access denied”, “Sign in” or “Verify” each marks one of them. When a board returns one of them, the options are a rendered fetch through residential proxies (15 credits a page), the board’s own structured endpoint where one exists (Indeed Job API and Glassdoor Job API return a posting as JSON for 5 credits each), or a lower request rate with a pause between pages. Two seconds between pages was enough on ZipRecruiter for the run in this guide.

Inconsistent Data Formats

Salary, location, and date formats vary between postings. One job shows “$80K - $100K/yr”, another shows “$80,000 - $100,000 per year”.

The problem. Filtering and sorting becomes difficult when salary formats differ.

The solution. Parse text into standardized formats. For salary:

def parse_salary(salary_text):
    """Convert salary text to numbers"""
    if not salary_text or salary_text == 'N/A':
        return None
    
    # Remove commas and spaces
    text = salary_text.replace(',', '').replace(' ', '')
    
    # Find numbers
    numbers = re.findall(r'\d+', text)
    if not numbers:
        return None
    
    # Check if yearly or hourly
    multiplier = 1000 if 'K' in text else 1
    
    return {
        'min': int(numbers[0]) * multiplier,
        'max': int(numbers[1]) * multiplier if len(numbers) > 1 else None,
        'period': 'yearly' if '/yr' in text else 'hourly'
    }

Now you can filter jobs by minimum salary or calculate averages across postings.

Expired or Filled Positions

Job boards keep old listings online even after positions are filled. You might scrape jobs that are no longer accepting applications.

The problem. Your dataset includes outdated jobs that waste time when applying.

The solution. Filter by the “New” badge or posting date when available. Some sites show “Posted 2 days ago” text that you can parse. Set a maximum age threshold.

Check for “actively recruiting” or “urgently hiring” badges that indicate the position is still open.

Legal Considerations for Job Scraping 

Job boards contain publicly posted information, but scraping them still has legal boundaries. Scraping job postings for personal job search or internal business analysis is typically acceptable. You’re automating what you could do manually by browsing listings.

What you can do:

• Scrape for personal job search and tracking applications
• Collect data for internal salary research or market analysis
• Monitor competitor hiring patterns for business intelligence

What you should not do:

• Republish scraped job listings on your own website
• Sell scraped job data to third parties
• Scrape or store personal information about job seekers

Before scraping, check the site’s robots.txt file and read their terms of service. Use delays between requests and scrape during off-peak hours. If you’re sharing insights publicly, aggregate the data and remove identifying details. You can say “Python developer salaries average $120K” without republishing specific listings.

If you’re unsure about what’s allowed, contact the job board or check if they offer an official API.

Conclusion

The ZipRecruiter scraper above transfers to any job site. Read the JobPosting markup on the posting page first, find the container elements holding job cards on the search page, identify the selectors for title, company, location, and salary, then loop through pages using their pagination structure.

Every job board works the same way. They all display listings in cards or rows, publish the same schema.org fields for Google Jobs, and paginate results. The HTML structure differs, but the extraction process stays identical. Inspect the page, read the markup, map the selectors for what the markup lacks, handle pagination.

Use this scraper as a template. When you need data from Indeed, LinkedIn, or any other job board, follow the same steps. Check if they use JavaScript rendering and choose HasData API or basic requests accordingly. Build the scraper incrementally, test with one page first, then scale to multiple pages.